From owner-mpi-pt2pt@CS.UTK.EDU Mon Nov 23 14:37:41 1992 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA18480; Mon, 23 Nov 92 14:37:41 -0500 Received: by CS.UTK.EDU (5.61++/2.8s-UTK) id AA23460; Mon, 23 Nov 92 14:28:46 -0500 Received: from THUD.CS.UTK.EDU by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA23454; Mon, 23 Nov 92 14:28:42 -0500 Received: from LOCALHOST.cs.utk.edu by thud.cs.utk.edu with SMTP (5.61++/2.7c-UTK) id AA06042; Mon, 23 Nov 92 14:28:41 -0500 Message-Id: <9211231928.AA06042@thud.cs.utk.edu> To: mpi-pt2pt@cs.utk.edu Cc: wade@cs.utk.edu Subject: a test Date: Mon, 23 Nov 92 14:28:40 EST From: Reed Wade this is a test of the mpi-pt2pt mailing list. -reed Mon Nov 23 16:03:12 EST 1992 From owner-mpi-pt2pt@CS.UTK.EDU Tue Nov 24 23:08:30 1992 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA15646; Tue, 24 Nov 92 23:08:30 -0500 Received: by CS.UTK.EDU (5.61++/2.8s-UTK) id AA25775; Tue, 24 Nov 92 22:28:54 -0500 Received: from Mail.Think.COM by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA25771; Tue, 24 Nov 92 22:28:51 -0500 Received: from Electryon.Think.COM by mail.think.com; Tue, 24 Nov 92 22:28:43 -0500 From: Adam Greenberg Received: by electryon.think.com (4.1/Think-1.2) id AA20255; Tue, 24 Nov 92 22:28:41 EST Date: Tue, 24 Nov 92 22:28:41 EST Message-Id: <9211250328.AA20255@electryon.think.com> To: mpi-pt2pt@cs.utk.edu Subject: Nonblocking functions and handlers. I would like to float a few questions about what we mean by the modifier nonblocking it is applied to the MPI specification. Nonblocking Send: This is the curious one. Does is mean: a) The user may alter the contents of the buffer once the send function returns control? or b) The user must check the status of the send operation to determine when it is safe to alter the contents of the buffer? Nonblocking receive: This seems pretty clear. The user must inquire of the system to determine if the user specified buffer contains the desired message. In either case, it may be desirable to specify an action that is to occur once the operation is complete. Consider adding two arguments to the send and receive functions: MPI__n( int , int tag, void * buffer, int len, void (* handler)(), void * arg ) Here, handler is a pointer to a function which accepts two arguments: handler( int msg_id, * arg ) Upon completion of the send or receive operation, the handler function is invoked and provided with the id of the message operation just completed and the argument address provided by the user. I raise the issue of handlers because we have ben asked repeatedly to provide such fuctionality. Thoughts? moose From owner-mpi-pt2pt@CS.UTK.EDU Wed Nov 25 10:38:02 1992 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA17471; Wed, 25 Nov 92 10:38:02 -0500 Received: by CS.UTK.EDU (5.61++/2.8s-UTK) id AA06314; Wed, 25 Nov 92 10:13:00 -0500 Received: from timbuk.cray.com by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA06310; Wed, 25 Nov 92 10:12:58 -0500 Received: from teak18.cray.com by cray.com (4.1/CRI-MX 2.2) id AA08435; Wed, 25 Nov 92 09:12:54 CST Received: by teak18.cray.com id AA05289; 4.1/CRI-5.6; Wed, 25 Nov 92 09:12:52 CST From: par@teak.cray.com (Peter Rigsbee) Message-Id: <9211251512.AA05289@teak18.cray.com> Subject: Re: Nonblocking functions and handlers. To: mpi-pt2pt@cs.utk.edu Date: Wed, 25 Nov 92 9:12:49 CST Cc: par@teak.cray.com (Peter Rigsbee) In-Reply-To: <9211250328.AA20255@electryon.think.com>; from "Adam Greenberg" at Nov 24, 92 10:28 pm X-Mailer: ELM [version 2.3 PL11b-CRI] Adam Greenberg writes: > In either case, it may be desirable to specify an action that is to occur once > the operation is complete. Consider adding two arguments to the send and > receive functions: > > MPI__n( int , int tag, void * buffer, int len, > void (* handler)(), void * arg ) > > Here, handler is a pointer to a function which accepts two arguments: > handler( int msg_id, * arg ) > > Upon completion of the send or receive operation, the handler function is > invoked and provided with the id of the message operation just completed and > the argument address provided by the user. I've never really liked this approach. I think it adds complexity without really adding much value. A couple of issues: - first, like most signal handling, this kind of capability tends to be more oriented towards C programs and away from Fortran programs (note that your prototype for "handler" is such that it could not be programmed in Fortran) - second, even if you have a signal handler, the mainline of the application must still poll something to see if the operation has completed; the difference is that the "something" if often a local variable rather than the messaging system. But the logic in the program is similar and it is likely that the number and decision of where to poll is driven by the algorithm. - third, do all architectures provide the user-level interrupts needed to implement this? If one doesn't, adding this functionality may simply not be possible. IMO, this seems like something that isn't desirable as a part of MPI1. - Peter Rigsbee From owner-mpi-pt2pt@CS.UTK.EDU Wed Nov 25 10:40:02 1992 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA17537; Wed, 25 Nov 92 10:40:02 -0500 Received: by CS.UTK.EDU (5.61++/2.8s-UTK) id AA06692; Wed, 25 Nov 92 10:30:22 -0500 Received: from antares.mcs.anl.gov by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA06686; Wed, 25 Nov 92 10:30:19 -0500 Received: from godzilla.mcs.anl.gov by antares.mcs.anl.gov (4.1/SMI-GAR) id AA03147; Wed, 25 Nov 92 09:28:46 CST From: gropp@antares.mcs.anl.gov (William Gropp) Received: by godzilla.mcs.anl.gov (4.1/GeneV4) id AA12985; Wed, 25 Nov 92 09:28:43 CST Date: Wed, 25 Nov 92 09:28:43 CST Message-Id: <9211251528.AA12985@godzilla.mcs.anl.gov> To: moose@think.com Cc: mpi-pt2pt@cs.utk.edu In-Reply-To: Adam Greenberg's message of Tue, 24 Nov 92 22:28:41 EST <9211250328.AA20255@electryon.think.com> Subject: Nonblocking functions and handlers. I'd love to see a standard implementation that invoked a routine on receipt, particularly if that routine was not restricted in any way (for example, restricted to not using any MPI calls). Unfortunately, doing this right makes demands on the underlying OS that may be more than some vendors are willing to commit to. We need to be careful here. Note that Unix does not provide this functionality to users, though VMS has for a long time. (Warning: radical position that I'm not sure even I hold follows:) An interesting issue is whether we should defer all nonblocking communications to a thread-based execution model. Bill Gropp From owner-mpi-pt2pt@CS.UTK.EDU Wed Nov 25 11:07:46 1992 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA19010; Wed, 25 Nov 92 11:07:46 -0500 Received: by CS.UTK.EDU (5.61++/2.8s-UTK) id AA07267; Wed, 25 Nov 92 11:02:46 -0500 Received: from NA-GW.CS.YALE.EDU by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA07263; Wed, 25 Nov 92 11:02:43 -0500 Received: from YOGI.NA.CS.YALE.EDU by CASPER.NA.CS.YALE.EDU via SMTP; Wed, 25 Nov 1992 11:02:31 -0500 Received: by YOGI.NA.CS.YALE.EDU (Sendmail-5.65c/res.client.cf-3.5) id AA26484; Wed, 25 Nov 1992 11:02:30 -0500 Date: Wed, 25 Nov 1992 11:02:30 -0500 From: berryman-harry@CS.YALE.EDU (Harry Berryman) Message-Id: <199211251602.AA26484@YOGI.NA.CS.YALE.EDU> To: mpi-pt2pt@cs.utk.edu, par@teak.cray.com Subject: Re: Nonblocking functions and handlers. Peter Rigsbee says: - first, like most signal handling, this kind of capability tends to be more oriented towards C programs and away from Fortran programs (note that your prototype for "handler" is such that it could not be programmed in Fortran) Although the prototype given by Greenberg precludes the use of Fortran, the approach certainly does not preclude Fortran. Obviously Fortran can pass pointers to functions as parameters. In particular, the iPSC series (including the Delta) support a similar mechanism in Fortran through the hrecv(), et al. -scott berryman Yale Computer Science Department From owner-mpi-pt2pt@CS.UTK.EDU Wed Nov 25 11:38:31 1992 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA19507; Wed, 25 Nov 92 11:38:31 -0500 Received: by CS.UTK.EDU (5.61++/2.8s-UTK) id AA07714; Wed, 25 Nov 92 11:22:36 -0500 Received: from timbuk.cray.com by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA07710; Wed, 25 Nov 92 11:22:33 -0500 Received: from teak18.cray.com by cray.com (4.1/CRI-MX 2.2) id AA13731; Wed, 25 Nov 92 10:22:31 CST Received: by teak18.cray.com id AA05599; 4.1/CRI-5.6; Wed, 25 Nov 92 10:22:30 CST From: par@teak.cray.com (Peter Rigsbee) Message-Id: <9211251622.AA05599@teak18.cray.com> Subject: Re: Nonblocking functions and handlers. To: mpi-pt2pt@cs.utk.edu Date: Wed, 25 Nov 92 10:22:26 CST In-Reply-To: <199211251602.AA26484@YOGI.NA.CS.YALE.EDU>; from "Harry Berryman" at Nov 25, 92 11:02 am X-Mailer: ELM [version 2.3 PL11b-CRI] Scott Berryman writes: > - first, like most signal handling, this kind of capability tends > to be more oriented towards C programs and away from Fortran > programs (note that your prototype for "handler" is such that > it could not be programmed in Fortran) > > Although the prototype given by Greenberg precludes the use of > Fortran, the approach certainly does not preclude Fortran. > Obviously Fortran can pass pointers to functions as parameters. Point well taken. But there are two aspects of the interface to consider. Fortran passing pointers to functions as parameters is fine. But a Fortran programmer might also want to write the handler in Fortran, which would then require that the handler's arguments all be pointers to things. But this would bother C programmers and would be inconsistent with other "signal handlers". > In particular, the iPSC series (including the Delta) support > a similar mechanism in Fortran through the hrecv(), et al. My iPSC documentation (which may be out of date) shows that hrecv() can be called by Fortran but requires that the handler be written in C: proc is the procedure executed when the receive is complete. This procedure must be written by the user and must be a C procedure with four arguments. I'd argue that having to write the handler in C makes this less useful to Fortran programmers, especially if you are trying to write something portable. There is no real standard defining how C and Fortran interact. The C handler could call a subhandler written in Fortran, but this is getting kind of messy for something that should be simple. - Peter Rigsbee From owner-mpi-pt2pt@CS.UTK.EDU Wed Nov 25 13:43:10 1992 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA21772; Wed, 25 Nov 92 13:43:10 -0500 Received: by CS.UTK.EDU (5.61++/2.8s-UTK) id AA10110; Wed, 25 Nov 92 13:30:11 -0500 Received: from Mail.Think.COM by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA10106; Wed, 25 Nov 92 13:30:09 -0500 Received: from Electryon.Think.COM by mail.think.com; Wed, 25 Nov 92 13:30:05 -0500 From: Adam Greenberg Received: by electryon.think.com (4.1/Think-1.2) id AA22742; Wed, 25 Nov 92 13:30:04 EST Date: Wed, 25 Nov 92 13:30:04 EST Message-Id: <9211251830.AA22742@electryon.think.com> To: berryman-harry@CS.YALE.EDU Cc: mpi-pt2pt@cs.utk.edu, par@teak.cray.com In-Reply-To: <199211251602.AA26484@YOGI.NA.CS.YALE.EDU> "berryman-harry@CS.YALE.EDU" Subject: Nonblocking functions and handlers. Date: Wed, 25 Nov 1992 11:02:30 -0500 From: berryman-harry@CS.YALE.EDU (Harry Berryman) Peter Rigsbee says: - first, like most signal handling, this kind of capability tends to be more oriented towards C programs and away from Fortran programs (note that your prototype for "handler" is such that it could not be programmed in Fortran) Although the prototype given by Greenberg precludes the use of Fortran, the approach certainly does not preclude Fortran. Obviously Fortran can pass pointers to functions as parameters. In particular, the iPSC series (including the Delta) support a similar mechanism in Fortran through the hrecv(), et al. -scott berryman Mea culpa. The prototype was not meant to exclude Fortran (in fact we through a discussion of this issue here last week). The handler should viewed as taking two pointers: void handler( int * msg_id, * arg ) Sorry for any confusion: moose From owner-mpi-pt2pt@CS.UTK.EDU Wed Nov 25 13:44:47 1992 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA21797; Wed, 25 Nov 92 13:44:47 -0500 Received: by CS.UTK.EDU (5.61++/2.8s-UTK) id AA09803; Wed, 25 Nov 92 13:14:38 -0500 Received: from relay2.UU.NET by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA09784; Wed, 25 Nov 92 13:14:29 -0500 Received: from uunet.uu.net (via LOCALHOST.UU.NET) by relay2.UU.NET with SMTP (5.61/UUNET-internet-primary) id AA22170; Wed, 25 Nov 92 13:14:31 -0500 Received: from kailand.UUCP by uunet.uu.net with UUCP/RMAIL (queueing-rmail) id 131340.4011; Wed, 25 Nov 1992 13:13:40 EST Received: from brisk.kai.com (brisk) by kailand.kai.com via SMTP (5.65d-92031301) id AA12688; Wed, 25 Nov 1992 12:06:04 -0600 Received: by brisk.kai.com (920330.SGI-92101201) id AA08958; Wed, 25 Nov 92 12:06:02 -0600 Date: Wed, 25 Nov 92 12:06:02 -0600 Message-Id: <9211251806.AA08958@brisk.kai.com> To: mpi-pt2pt@cs.utk.edu, mpi-collcomm@cs.utk.edu, mpi-formal@cs.utk.edu, mpi-ptop@cs.utk.edu Reply-To: William.Gropp's.message.of.Wed@kai.com, 25 Nov 92 09:28:43 CST <9211251528.AA12985@godzilla.mcs.anl.gov> Subject: Nonblocking functions and handlers. From: Steven Ericsson Zenith Sender: zenith@kai.com Organization: Kuck and Associates, Inc. 1906 Fox Drive, Champaign IL USA 61820-7334, voice 217-356-2288, fax 217-356-5199 Bill Gropp writes: (Warning: radical position that I'm not sure even I hold follows:) An interesting issue is whether we should defer all nonblocking communications to a thread-based execution model. I'm not so sure this is a radical position Bill since even nonsynchronized communication will need to be defined formally this way. Nonsynchronized communication is in effect creating a parallel process that has the job of passing the communication on. Al Geist earlier asked the question wheather buffers used by nonsynchronized communication should be accessible after the communication has started - the answer should be - no, unless by some explicit mechanism that formally amounts to a communication with the process mentioned above. Any nonexplicit interaction (e.g. a write to the buffer) would have to be specified as formally equivalent to an explicit interaction. Also, there is quite a range of terminology in use. One common error: "Asynchronous" and "synchronous" has quite a particular meaning in EE and when CS people use the terms in relation to message passing they usually mean NONSYNCHRONIZED and SYNCHRONIZED. Also BLOCKING = SYNCHRONIZED. Let us begin a glossary that defines the terms we use - if no-one else volunteers I'll take this to be the responsibility of the Formal Specification Subcommittee. So I'm looking for volunteers from that subcommittee. Steven From owner-mpi-pt2pt@CS.UTK.EDU Sat Nov 28 19:07:59 1992 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA08243; Sat, 28 Nov 92 19:07:59 -0500 Received: by CS.UTK.EDU (5.61++/2.8s-UTK) id AA22503; Sat, 28 Nov 92 18:46:34 -0500 Received: from cunyvm.cuny.edu by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA22499; Sat, 28 Nov 92 18:46:32 -0500 Message-Id: <9211282346.AA22499@CS.UTK.EDU> Received: from YKTVMV by CUNYVM.CUNY.EDU (IBM VM SMTP V2R2) with BSMTP id 6099; Sat, 28 Nov 92 18:45:57 EST Date: Sat, 28 Nov 92 18:35:53 EST From: "Marc Snir" X-Addr: (914) 945-3204 (862-3204) 28-226 IBM T.J. Watson Research Center P.O. Box 218 Yorktown Heights NY 10598 To: mpi-pt2pt@cs.utk.edu Subject: Our work Reply-To: SNIR@watson.ibm.com First, a technical issue: If somebody cannot print postscript, or has strong preferences for Latex, pls let me know. Now for content. I would like to use the outline I just sent as a working document for our subcommittee, and modify it periodically to reflect input I receive from the remaining group members. I shall update this document every weekend, or every 2 weekends, according to the number of comments I get, and the amount of spare time my family grants me. Of course, decisions will be taken when we meet, but I feel it is useful to have a working document that reflects in a somewhat coherent manner the state of our deliberations. So please, start sending your comments to mpi-pt2pt, assuming the next checkpoint is end of next week. I want to replace as soon as possible the I's by We's. From owner-mpi-pt2pt@CS.UTK.EDU Thu Dec 3 10:39:15 1992 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA01501; Thu, 3 Dec 92 10:39:15 -0500 Received: by CS.UTK.EDU (5.61++/2.8s-UTK) id AA16013; Thu, 3 Dec 92 10:20:18 -0500 Received: from gmdzi.gmd.de by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA16008; Thu, 3 Dec 92 10:20:13 -0500 Received: from f1neuman.gmd.de (f1neuman) by gmdzi.gmd.de with SMTP id AA25545 (5.65c/IDA-1.4.4 for ); Thu, 3 Dec 1992 16:19:46 +0100 Received: by f1neuman.gmd.de id AA14744; Thu, 3 Dec 1992 16:19:30 +0100 Date: Thu, 3 Dec 1992 16:19:30 +0100 From: Rolf.Hempel@gmd.de Message-Id: <9212031519.AA14744@f1neuman.gmd.de> To: mpi-pt2pt@cs.utk.edu Subject: Marc Snir's paper Cc: gmap10@f1neuman.gmd.de I would like to make some comments on the document Marc Snir has distributed. 1. I agree on all points in the list of goals. I find it especially important to define something which is not too far away from common practice, otherwise nobody would follow us. A point which is missing on the list (perhaps because it is too obvious): the interface should not contain more functions than are really necessary. We must avoid to end up with a mixed bag which contains multiple mechanisms for the same thing. 2. I like Marc's typification scheme for message passing functions based on the 4 suboperations. I understand those suboperations more as a way to describe the semantics rather than the implementation of the (compound) functions which the user will see. I'm not sure that the channel based operations are necessary on the user level. Basically there are the two addressing schemes for message passing: either a message is sent to a named process, or it is sent along a named connection (channel), in which case the processes don't even need to have identifiers. We had a long discussion here in Europe on which scheme is better. In the end it turns out that both are more or less equivalent. Most existing user interfaces tend towards the named process approach. The only reason for supporting both concepts in the same interface seems to be the possibility of some additional optimization. If suboperations (1) and (4) are separated, ressource allocation and deallocation can be saved. However, I think that a clever operating system can detect frequently used communication paths automatically, and it can establish a permanent link for them. (Link caching) 3. We will have various message passing mechanisms, especially synchronous and asynchronous communication (global/local send termination). I strongly recommend that the corresponding send and receive functions must match. On one hand this gives the opportunity for some implementation optimization, and also it might be used for correctness checking. 4. As Marc said, in MPI1 termination is either local or global, at the user's choice. I really think that this is necessary, and a restriction on either synchronous or asynchronous communication would be hard to accept. 5. Marc suggests that there should be no "unsuccessful return" of a message passing operation. I agree. 6. On page 7, Marc discusses the different protocols for message passing. I'm not sure that we want to define this in detail, since it is very much implementation dependent. On the other hand we will have to accept the fact that a program will run on a machine with huge system buffers and not on a smaller system. Somehow this is true with other system resources, also. 7. Page 7, further down: if we deal with heterogeneous networks, a contiguous message buffer may contain sections with different data type. The reason is always that one wants to send as few messages as possible (startup minimization). Personally I prefer to have a separate library call before the send which specifies the contents of the message buffer. This information is then used or ignored by the send routine depending on the data representations in the sender and receiver processors. 8. If our goal is to define a scalable interface, setting up groups by specifying member lists in all participating processes is a problem. This is especially so for the handling of very regular group structures, like grids or tori, since these will most likely be used by applications with very large process numbers. I think the Virtual Topology subcommittee has to deal with this problem. 9. Communication context: I very much favor option 1. (i.e. context switch by some special functions) against explicitely specifying the context in a (collective) communication statement. I do not see that the explicit specification would lead to less user errors, and it would add another item to the argument lists. 10. Receive criteria: I do not see a good point for having "don't care" addresses in sends. For "don't care" in receive I prefer named constants over "magic values". Be sure not to use arguments for both input to and output from a function, especially if it may contain wildcards on input. There is no better way to make sure your constants get overwritten! 11. Matching of data: In Fortran I think option 2. is acceptable. If the receive buffer is too short, this is a user error. In C there is another option which should be supported by the interface: if the receive buffer pointer is NULL, the system allocates one with the right size automatically. 12. Syntax: multiple modes like in MPI1 are okay with me, but I hate to see character strings there. Talk to any implementor who cares about efficiency, and you know what I mean. 13. Also, in Fortran I find it better to use procedure calls for most functions. 14. Miscelania: a) We will need a nonblocking AWAIT operation. This is a great help in organizing the overlap of computation and communication. b) How could we survive without a PROBE function? c) What about providing an inquiry subroutine, which returns the message length, the sender, and the type of the "last message" which was received? For non-blocking messages this is not well defined, but a solution should be possible. This has become a longer message than I had planned. Sorry for bothering you all so much! Rolf Hempel From owner-mpi-pt2pt@CS.UTK.EDU Wed Dec 9 16:39:48 1992 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA21532; Wed, 9 Dec 92 16:39:48 -0500 Received: by CS.UTK.EDU (5.61++/2.8s-UTK) id AA22026; Wed, 9 Dec 92 16:32:54 -0500 Received: from antares.mcs.anl.gov by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA22022; Wed, 9 Dec 92 16:32:51 -0500 Received: from godzilla.mcs.anl.gov by antares.mcs.anl.gov (4.1/SMI-GAR) id AA05364; Wed, 9 Dec 92 15:32:41 CST From: gropp@antares.mcs.anl.gov (William Gropp) Received: by godzilla.mcs.anl.gov (4.1/GeneV4) id AA00525; Wed, 9 Dec 92 15:32:39 CST Date: Wed, 9 Dec 92 15:32:39 CST Message-Id: <9212092132.AA00525@godzilla.mcs.anl.gov> To: mpi-pt2pt@cs.utk.edu Subject: Marc Snir's paper Cc: lusk@antares.mcs.anl.gov Thanks to Marc for laying out many of the issues so clearly. We find that we agree with many and disagree with some. Goals: (1) We think that those most in need of a standard are those writing libraries, and therefor the standard should cater to their needs as well as those of end users. It will be easy to design simple application programming interfaces on top of the MPI standard. (2) We strongly agree with this point (allow efficient communication). (3) We are not sure what "allow (but no mandate)" for heterogeneous communications means. If this means that vendors of homogeneous massively parallel systems may implement only the homogeneous operations, then we agree, as long as the heterogeneous operations are defined by the MPI standard. We agree in principle with the others. Framework: We also find the described framework useful for thinking about the operations, but not necessarily the right structure for the MPI. Issues: An exchange operation can be useful. At least two MPP vendors (TMC and Intel) provide such operations. We also agree an leaving out RPC, active messages, etc, as these are topics of rapidly moving research. We believe that there may be something more inclusive than channels that can serve as the basis for specifying high-performance communication operations. We do not find the discussion of "local versus global" termination to be a good way to consider the issues of blocking, nonblocking, and synchronous communication. In particular, blocking and nonblocking refer to the state of the buffer containing the message on return from the routine (be it send, receive, or wait), and synchronous (in the MPI proposal) really refers to an alternate protocol for the delivery of a message. We believe that relying on an exception handling mechanism for error handling will make the system too fragile. We propose allowing the return of error conditions. An alternate proposal allows the user to specify an error routine to be called in the event of an error. Note that library implementors may need to handle errors explicitly. Correctness Criteria: We believe that it is important to allow the user to determine (at least a bound on) the amount of resources available. Further, it MAY be appropriate to specify a minimum size that is supported (otherwise zero may be consistent with the letter of the standard). Communication Parameters: Our draft implementation includes an extension to the current proposal that permits the communication of messages of mixed datatype (we interpret "type" on page 7 to mean "datatype" rather than "message type"). This is a simple extension to the data description vector that allows for heterogeneous communication. Groups and context: We agree that the collective communication committee should be the one to discuss these issues. We do feel that the destination and source processor ids should not be relative to some "current" group. Note that such a functionality (with some restrictions) can be provided by a simple layer on top of MPI. For an example of a problem with making processor id's relative to a current "group", consider recv(source=DONT_CARE,...). Does this mean only in the current group? What is the result of MPI_INFOS() on the received message? What if two processors A and B are both in two groups alpha and beta, but with different rankings (ie., A = 2 and B = 5 in alpha, and A = 12 and B = 0 in beta). Now A (current group alpha) sends to B (current group beta). B does recv(source=DONT_CARE,...). Does B receive this message? What is the value of MPI_INFOS()? If B doesn't receive this message, then how is a communication context different from a group that is a copy of the current group (which would seem to supply the same semantics)? Matching of send and receive: We believe that there should be two types of matching operations: a simple (efficient) one on message type, and a general one using a user-defined predicate (matching function). Our experience has been that once the users have "select by source", they'll then want "select by source range and/or type range and/or message size and/or ...". We also agree that truncation of a received message into a too-short buffer should be an error (or at least detectable!). Syntax: We agree with multiple function names rather than multiple modes. It is easier to provide a efficient implementation and clear semantics in this case (for example, having the return code always have the same meaning independent of the input arguments). Note also that it is easy, given a multiple function names approach, to produce one-routine-does-everything interfaces but not vice-versa. Miscelania: We must have a non-blocking AWAIT. We'd like to AWAIT several messages. We must have a PROBE. We agree that the issue of returning data on received messages (particularly those done with nonblocking operations) is very important. One approach (used by one MPP vendor) is to ask for this info by handle, and then explicitly free the handle. This eliminates unnecessary global state (the currently most recently received message) from the semantics, and makes it possible to easily describe what exactly is returned by these "INFO" routines. Bill Gropp and Rusty Lusk From owner-mpi-pt2pt@CS.UTK.EDU Wed Dec 9 17:09:55 1992 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA22659; Wed, 9 Dec 92 17:09:55 -0500 Received: by CS.UTK.EDU (5.61++/2.8s-UTK) id AA22401; Wed, 9 Dec 92 16:51:16 -0500 Received: from super.super.org by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA22397; Wed, 9 Dec 92 16:51:13 -0500 Received: from b125 (b125.super.org) by super.super.org (4.1/SMI-4.1) id AA08797; Wed, 9 Dec 92 16:51:11 EST Received: by b125 (4.1/SMI-4.1) id AA04506; Wed, 9 Dec 92 16:51:10 EST Date: Wed, 9 Dec 92 16:51:10 EST From: lederman@b125.super.org (Steve Lederman) Message-Id: <9212092151.AA04506@b125> To: mpi-pt2pt@cs.utk.edu Subject: thoughts I have a few thoughts on the ideas that have been passed around. I'm sorry for not responding sooner but a cold has slowed me down. First, on Marc's document of Nov. 28: - Goal 4 is for a C, C++, F77 and F90 binding. The draft Intro that Jack sent around on Nov. 30 states in the subsection for Who Should Use This Standard? that it is for Fortran 77 and/or C. Do we wish to include C++ and F90 which would likely to have different interfaces? - I find the SEND_RECV operation an interesting option. On the old Mark/?? machines from Caltech/JPL there was a call (if my recollection is correct) that allowed for the same user buffer to serve as the send and receive buffer. I assume this is not what is intended here? I think SEND_RECV would be used with some frequency by users. Would it simply be a convenience or can it be made faster than issuing the two calls. I would think it would be faster on some architectures (even more than the two call overhead). If it is faster then my inclination is to include it even though it adds another call. comments? - I have a question regarding when a communication operation is completed. On a send, does it terminate locally if the user buffer is copied into a system buffer but has not left the node? (I believe this is done on the CM5.) Analogously, does it terminate globally if it gets to a system buffer on the receiving node or until it is in the user buffer? As for choosing local, global or both I vote for the current choice of both. I can see applications for both. - I think that Marc's suggestion that MPI functions not return unsuccessfully has merit. It would be pain to be constantly checking the status after each operation to be safe. Along these lines is what to do if the send and recv message lengths are different. The current standard allows for fewer bytes to be received and the recv returns the actual number. Again, I think it would be a pain to have to check every time. If this is a desirable feature to allow different lengths, then I vote to make it users selectable. I know I have found many bugs by not having messages that were a different size being received. - I think that a PROBE function would be a good idea. Given that the standard is not currently going to support active messages, etc. it seems that being able periodically check if any messages have arrived would be a useful feature. Another thought: At the meeting in Minneapolis, several people, including myself, discussed the advantage of having a send where the sending process knows the receiving process has already posted a receive. This could relieve the send process of having to check with the receiving process since it knows it will not have to buffer the message in system space and it knows it is ready to receive the message. If a system could not take advantage of this then it could revert to a normal send. That is, the standard would not specify the way the send would occur, it would allow for an alternative protocol if the vendor wished. This is the rr (receiver ready) option in Bill Gropp and Ewing Lusk's implementation. I would like to propose we include this in the standard. Thoughts? Well enough for one message. Steve From owner-mpi-pt2pt@CS.UTK.EDU Mon Dec 14 15:48:42 1992 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA24886; Mon, 14 Dec 92 15:48:42 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA17233; Mon, 14 Dec 92 15:48:21 -0500 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Mon, 14 Dec 1992 20:48:20 GMT Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from THUD.CS.UTK.EDU by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA17227; Mon, 14 Dec 92 15:48:19 -0500 From: Jack Dongarra Received: by thud.cs.utk.edu (5.61++/2.7c-UTK) id AA03749; Mon, 14 Dec 92 15:48:17 -0500 Date: Mon, 14 Dec 92 15:48:17 -0500 Message-Id: <9212142048.AA03749@thud.cs.utk.edu> To: mpi-collcomm@cs.utk.edu, mpi-pt2pt@cs.utk.edu Subject: Re: Message Passing Interface Forum Forwarding: Mail from '"Dr. C.D. Wright" ' dated: Mon, 14 Dec 92 12:16:10 GMT ---------- Begin Forwarded Message ---------- >From @ibm.liv.ac.uk:CDW10@LIVERPOOL.AC.UK Mon Dec 14 07:20:05 1992 Return-Path: <@ibm.liv.ac.uk:CDW10@LIVERPOOL.AC.UK> Received: from mail.liv.ac.uk by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA22696; Mon, 14 Dec 92 07:19:55 -0500 Received: from ibm.liverpool.ac.uk by mailhub.liverpool.ac.uk via JANET with NIFTP (PP) id <21042-0@mailhub.liverpool.ac.uk>; Mon, 14 Dec 1992 12:19:28 +0000 Received: from UK.AC.LIVERPOOL by MAILER(4.4.t); 14 Dec 1992 12:20:02 GMT Date: Mon, 14 Dec 92 12:16:10 GMT From: "Dr. C.D. Wright" Subject: Re: Message Passing Interface Forum To: dongarra@edu.utk.cs Message-Id: <"mailhub.li.044:14.11.92.12.19.28"@liverpool.ac.uk> Status: RO Hi. Since I am in the UK it is clear that I can't actively participate in the MPI Forum. I do, however, have one particular problem with every comms library I have used so far that I would like to see addressed in any new "standard", and I hope you can pass this on to whoever is the appropriate person to deal with it. In many packages such as PVM, PARMACS, p4, etc, it is possible to probe for and/or receive messages selectively, the selection being based on the message type (usually in integer) and/or the sender. This is overly restrictive. It would be far more useful if the message's format were sufficiently well defined for the user to be able to provide their own selection function to be passed in and used as the basis for reception and/or probing. That's it. Hope you can do something with this gripe/suggestion. Colin. ----------- End Forwarded Message ----------- From owner-mpi-pt2pt@CS.UTK.EDU Tue Dec 15 19:28:34 1992 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA21803; Tue, 15 Dec 92 19:28:34 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA15587; Tue, 15 Dec 92 19:28:11 -0500 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Wed, 16 Dec 1992 00:28:10 GMT Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from helios.llnl.gov by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA15573; Tue, 15 Dec 92 19:28:06 -0500 Received: by helios.llnl.gov (4.1/LLNL-1.18) id AA11599; Tue, 15 Dec 92 16:30:03 PST Date: Tue, 15 Dec 92 16:30:03 PST From: tony@helios.llnl.gov (Anthony Skjellum) Message-Id: <9212160030.AA11599@helios.llnl.gov> To: dongarra@cs.utk.edu, mpi-collcomm@cs.utk.edu, mpi-pt2pt@cs.utk.edu Subject: Re: Message Passing Interface Forum That is what we have been talking about in Zipcode for a long time. - Tony From owner-mpi-pt2pt@CS.UTK.EDU Fri Dec 25 19:15:36 1992 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA20495; Fri, 25 Dec 92 19:15:36 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA09422; Fri, 25 Dec 92 19:15:07 -0500 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Sat, 26 Dec 1992 00:15:05 GMT Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from msr.EPM.ORNL.GOV by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA09400; Fri, 25 Dec 92 19:15:04 -0500 Received: by msr.EPM.ORNL.GOV (5.61/1.34) id AA14461; Fri, 25 Dec 92 19:14:59 -0500 Date: Fri, 25 Dec 92 19:14:59 -0500 From: geist@msr.EPM.ORNL.GOV (Al Geist) Message-Id: <9212260014.AA14461@msr.EPM.ORNL.GOV> To: mpi-pt2pt@cs.utk.edu Subject: point-to-point proposal consistent with collective proposal Point-to-point Communication Proposal ------------------------------------- After reading Marc Snir's Message Passing Interface Outline and the other point-to-point comments, I would like to propose the following minimum set of point-to-point routines for discussion at the January meeting in Dallas. Stated Goals: 1. Design an application programming interface. 2. Design an interface that is not too different from current practice 3. Define an interface that can be quickly implemented on many vendor platforms. 4. Focus on a proposal that can be agreed upon in 6 months. 5. Provide a reliable communication interface. 6. Allow efficient communication. 7. Interface should not contain more functions than are really necessary. ----------------------------------------------------------------------------- General Comments: The collective communication interface should be an extension of the point-to-point interface. a. As Marc points out on page 2 "SEND and RECV are a particular case of broadcast in a group of size 2; this observation can be used to check if the definition of collective communication semantics are consistent with the definition of point-to-point communication." b. Collective routines like broadcast should provide the same message data format (contiguous, buffer with stride, etc.) as point-to-point routines. c. By using a structured name space (described on pages 7-8 of [Snir]) where all processes are identified by a (group,rank) pair for both the point-to-point and collective routines, then the users will have a consistent naming scheme across all the MPI communication routines. And those desiring a flat name space can have it by using the default group "ALL". d. Syntax guidelines - short, mnemonic names; systematic ordering of parameters; systematic naming convention. ------------------------------------------------------------------------------- Proposed point-to-point routines: 1. info = MPI_SEND( buf, bytes, type, gid, dest ) Function: Sends a contiguous message of length "bytes" pointed to by "buf" to the process identified by ("gid","dest") with a message tag of "type". On return "info" contains the error code. Conceptually the user can view this function as "I want this message delivered to the specified process in the fastest reliable manner." Implementation comments: On return user can consider buf is safe for reuse. MPI_SEND need not be strictly blocking. Users do not want to have to dance around the send buffer with STAUS and WAIT checking for when the send buffer is safe to change. If a compiler can determine that (or rearrange code so that) the send buffer is not touched before the send completes, then the implementation could be non-blocking. If a multiprocessor requires synchronous communication or routing along a particular topology, then this could be set in a global routine that affects all point-to-point and collective routines. 2. info = MPI_RECV( buf, bytes, type, gid, source ) Function: Receive the message specified by "type", "gid", and "source" into "buf". The message must be <= "bytes" in size or "info" returns error. Routine waits until specified message has arrived [like blocking recv]. On return "info" contains the error code. A minus one (-1) in "type", "gid", or "source" means don't care and that argument will match any message. Conceptually the user can view this function as "I must have the specified message before I can do anything else. I will wait until it arrives." 3. info = MPI_NRECV( buf, bytes, type, gid, source ) Function: Checks to see if the specified message has arrived [like probe]. A minus one (-1) in "type", "gid", or "source" means don't care and that argument will match any message. If the message has arrived it is returned in "buf" and "info" = 1. The message must be <= "bytes" in size or "info" returns error. If the message has not arrived, then the routine returns with "info" = 0. Values less than 0 returned by "info" indicate an error. MPI_NRECV can be called multiple times to check on the arrival of a message, and the user can later call MPI_RECV to wait on the same message. Conceptually the user can view this function as "I would like to receive the specified message now, but if it is not here, I will not wait for it. I will go do something else." Implementation is not specified here, but for efficiency, implementors may wish to post a receive on the first call to MPI_NRECV and just do checks [like status] on repeat calls. If implemented like this, MPI_RECV must be able to check both the system queue and the posted queue. 4. type = MPI_TYPE() 5. bytes = MPI_BYTES() 6. gid = MPI_GID() 7. source = MPI_SOURCE() Function: These routines return the actual values for the last message returned by MPI_RECV or MPI_NRECV. Values < 0 indicate an error. Allows the user to determine the size of the received message and the values of "don't care" arguments. 8. bytes = MPI_PACK( data_type, nitems, stride, inbuf, outbuf ) 9. bytes = MPI_UNPACK( data_type, nitems, stride, inbuf, outbuf ) Function: General purpose routines for packing and unpacking an arbitrarily complex message buffer. Allows for non-contiguous messages, mixed data types in a single message, and heterogeneity between machines. Same routines can be used to build collective communication buffers, for example for MPI_BCAST. On return "outbuf" contains the (potentially) encoded data, and "bytes" is the size in bytes of the encoded data. "bytes" < 0 indicates an error. The following data types would be supported: BYTE INTEGER*2 INTEGER*4 REAL*4 REAL*8 COMPLEX*8 COMPLEX*16 I would also like to see the following data types considered: CHARACTER_STRING INTEGER*8 REAL*16 COMPLEX*32 "nitems" is the total number of items of this data type to pack or unpack from "inbuf". "stride" can be used for packing rows of a matrix in Fortran or columns of a matrix in C. MPI_PACK can be called multiple times to construct a single message buffer. MPI_UNPACKs must be called by the user to decode such a buffer on receipt. Neither of these routines need be called if the user does not require non-contiguous messages, mixed data types, or heterogeneity. Having separate packing and unpacking routines allows send and receive to be optimized for a contiguous stream of bytes, and simplifies future extensions of MPI into areas such as heterogeneity. =============================================================================== For completeness here is the present proposal from the collective communication subcommittee. 1. info = MPI_BCAST( buf, bytes, type, gid, root ) Function: Called by all members of the group "gid" using the same arguments for "bytes", "type", "gid", and "root". On return the contents of "buf" on "root" is contained in "buf" on all group members. On return "info" contains the error code. 2. info = MPI_GATHER( buf, bytes, type, gid, root ) Function: Called by all members of the group "gid" using the same arguments for "bytes", "type", "gid", and "root". On return all the individual "buf" are concatenated into the "root" buf, which must be of size at least gsize*bytes where gsize is group size. The data is laid in the "root" buf in rank order that is | gid,0 data | gid,1 data | ...| gid, root data | ...| gid, gsize-1 data | Other member's "buf" are unchanged on return. On return "info" contains the error code. 3. info = MPI_GLOBAL_OP( inbuf, bytes, type, gid, op, outbuf ) Function: Called by all members of the group "gid" using the same arguments for "bytes", "type", "gid", and "op". On return the "outbuf" of all group members contains the result of the global operation "op" applied pointwise to the collective "inbuf". For example, if the op is max and inbuf contains two float point numbers then outbuf(1) = global max( inbuf(1)) and outbuf(2) = global max( inbuf(2)) A set of standard operations are supplied with MPI including: global max - for each data type global min - for each data type global sum - for each data type global mult- for each data type global AND - for integer and logical type global OR - for integer and logical type global XOR - for integer and logical type Optionally the users may define their own global functions for this routine. On return "info" contains the error code. 4. info = MPI_SYNCH( gid ) Function: Called by all members of the group "gid" Returns only when all members have called this function. On return "info" contains the error code. 5. gid = MPI_MKGROUP( list_of_processes ) Function: Called by all processes in the list. Forms a logical group containing the listed processes and assigns each process a unique rank in the group. The ranks are consecutively numbered from 0 to gsize-1. On return "gid" is an MPI assigned group ID (or error code if < 0) 6. gsize = MPI_GROUPSIZE( gid ) Function: Can be called by any process. On return "gsize" is the number of members in the group "gid" (or error code if < 0). 7. rank = MPI_MYRANK( gid ) Function: Can be called only by members of group "gid". On return "rank" is the rank of the calling process in group "gid" (an integer between 0 and gsize-1) or error code if < 0. =========================================================================== From owner-mpi-pt2pt@CS.UTK.EDU Mon Jan 4 08:50:16 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA19230; Mon, 4 Jan 93 08:50:16 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA07603; Mon, 4 Jan 93 08:49:52 -0500 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Mon, 04 Jan 1993 13:49:47 GMT Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from marge.meiko.com by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA07594; Mon, 4 Jan 93 08:49:43 -0500 Received: from hub.meiko.co.uk by marge.meiko.com with SMTP id AA06801 (5.65c/IDA-1.4.4 for ); Mon, 4 Jan 1993 08:49:38 -0500 Received: from float.co.uk (float.meiko.co.uk) by hub.meiko.co.uk (4.1/SMI-4.1) id AA15719; Mon, 4 Jan 93 13:49:35 GMT Date: Mon, 4 Jan 93 13:49:35 GMT From: jim@meiko.co.uk (James Cownie) Message-Id: <9301041349.AA15719@hub.meiko.co.uk> Received: by float.co.uk (5.0/SMI-SVR4) id AA02659; Mon, 4 Jan 93 13:49:02 GMT To: mpi-pt2pt@cs.utk.edu Subject: Multifarious Content-Length: 6062 Gentlepeople, Firstly a general point ======================= I'd like to add an additional criterion to the goals of the message passing interface. 8) The implementation and semantics of the interface should remain simple and "obvious" when multiple "lightweight" threads share the same address space. I'd like this because 1) Lightweight thread libraries on the Posix model are becoming available on most workstations (networks of which are one of our targets). 2) There is a clear industry trend to having a few processors on a logically (and often physically) shared memory both in workstations (e.g. SparcStation 10), and servers (SparcServer 690, RS6000 ??). It would be foolish to go down a path which made exploitation of such machines hard. Undoubtedly this trend will also occur in future MPP machines (it would "only" take a software change to the code on the Paragon to make it be like this. After all it has all the hardware there already). As a consequence of this I'd like to remove as much hidden state as possible from the message passing system, so 1) I dislike the functions which return information about "the last message", since in a single process with multiple threads there can be many "last" messages. The suggestion that these operations be performed on descriptors returned from the probe is much preferable (and semantically cleaner). 2) I dislike the idea of setting options in a global routine, because there may be other threads performing operations at the same time as the options are changing, and the semantics becomes very hard to pin down. Secondly A few comments on Al Geist's proposal of Christmas Day =============================================================== 1. MPI_SEND I liked the separation in the original draft between the concepts of blocking (the buffer is now free) and synchronous (the message has been received at the other end) communications. I'd like to preserve both possibilities. 3. MPI_NRECV I don't understand how this is intended to work. Consider the following code. char buffer[100]; int i; for (i=0; i<10; i++) { if (&MPI_NRECV(buffer[10*i],10,1,1,1)) break; } Questions : 1) Into which elements of buffer will the receive occur ? Al's implementation note suggests just the first, so what then do the other (up to) 9 calls achieve ? Will they receive the next nine matching messages ? (this doesn't appear very intuitive for a "polling" routine) 2) Should I now do a blocking MPI_RECV as well ? Or does this only make sense if the MPI_NRECVs all fail ? But if they do fail, is the receive buffer still queued to receive a matching message when it becomes available ? (and if so how do I cancel the outstanding receive and get back ownership of my buffer ?) 4,5,6,7 See comments above about this style of working. 8,9 If we're only having one form of packing, then it should probably be the more general one with arrays of pointers and counts, since it can subsume (admittedly at some cost) the strided style. Conclusion ========== I prefer the original draft to Al's version. I'd like to change the mode argument in the original draft (see my mail to mpi-lang). Removing internal state ======================= I'd also like to remove the hidden state ("last message"), there are various ways of achieving this, none of which is exceptionally attractive. 1) Return a structure from MPI_RECV (and friends) This is fine in C, but not good in Fortran 77 (and may be more expensive than we want when you don't need any of the information). e.g. typedef struct MPI_REPLY { int status; int nbytes; int whoFrom; int tag; /* More ? */ } MPI_REPLY; { MPI_REPLY res; res = MPI_RECV(...); if (res.whoFrom == 0) ... etc ... } 2) Return results indirect through individual parameters (arguments for the Fortran inclined) This works, but is very error prone, particularly in Fortran when the arguments are INOUT. (We have an interface like this at the moment, and it does cause problems). 3) Return a tag on which queries can be made. This means that you must then have another call to the system to dispose of the tag (and associated system storage). This is unpleasant on a simple blocking receive when you don't want any of the information (since it's another call to the library). 4) Have additional (possibly optional) OUT arguments through which the results are returned if required. Using optional keyword arguments in F90 this works fine. C can be made to work more or less. F77 can't (within the standard). 5) Have one required argument which is a pointer to a reply structure like that used in 1) which is then filled in by the function. This can work in all the languages. In C you could also allow NULL to mean no such results are required. IMHO Solution 5 is best. If we wanted to (and we may) then we could disallow direct access to this structure and provide functions to pick it apart (these could be macros in C, inline functions in C++). This would avoid having to specify the contents of the structure and allow for future expansion. Comments on Steve Lederman's note ================================= > At the meeting in Minneapolis, several people, including myself, > discussed the advantage of having a send where the sending process > knows the receiving process has already posted a receive. We have this in our current message passing model. It has proved to be an awful mistake. 98% of user's don't know when to use it, and then complain that their code no longer works when they've used it incorrectly. Please let's not do this. (I'm speaking from experience here). See you all in Dallas. -- Jim James Cownie Meiko Limited 650 Aztec West Bristol BS12 4SD England Phone : +44 454 616171 FAX : +44 454 618188 E-Mail: jim@meiko.co.uk or jim@meiko.com From owner-mpi-pt2pt@CS.UTK.EDU Mon Jan 4 08:59:46 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA19312; Mon, 4 Jan 93 08:59:46 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA08031; Mon, 4 Jan 93 08:59:34 -0500 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Mon, 04 Jan 1993 13:59:32 GMT Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from marge.meiko.com by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA08019; Mon, 4 Jan 93 08:59:30 -0500 Received: from hub.meiko.co.uk by marge.meiko.com with SMTP id AA06920 (5.65c/IDA-1.4.4 for ); Mon, 4 Jan 1993 08:59:25 -0500 Received: from float.co.uk (float.meiko.co.uk) by hub.meiko.co.uk (4.1/SMI-4.1) id AA15744; Mon, 4 Jan 93 13:59:21 GMT Date: Mon, 4 Jan 93 13:59:21 GMT From: jim@meiko.co.uk (James Cownie) Message-Id: <9301041359.AA15744@hub.meiko.co.uk> Received: by float.co.uk (5.0/SMI-SVR4) id AA02661; Mon, 4 Jan 93 13:58:46 GMT To: mpi-pt2pt@cs.utk.edu Subject: Incorrect Example Content-Length: 311 Appologies, the example should (of course) read for (i=0; i<10; i++) { if (MPI_NRECV(&buffer[10*i],10,1,1,1)) break; } -- Jim James Cownie Meiko Limited 650 Aztec West Bristol BS12 4SD England Phone : +44 454 616171 FAX : +44 454 618188 E-Mail: jim@meiko.co.uk or jim@meiko.com From owner-mpi-pt2pt@CS.UTK.EDU Mon Jan 4 09:33:03 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA19844; Mon, 4 Jan 93 09:33:03 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA09262; Mon, 4 Jan 93 09:32:48 -0500 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Mon, 04 Jan 1993 14:32:47 GMT Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from marge.meiko.com by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA09242; Mon, 4 Jan 93 09:32:36 -0500 Received: from hub.meiko.co.uk by marge.meiko.com with SMTP id AA07241 (5.65c/IDA-1.4.4); Mon, 4 Jan 1993 09:32:32 -0500 Received: from float.co.uk (float.meiko.co.uk) by hub.meiko.co.uk (4.1/SMI-4.1) id AA15874; Mon, 4 Jan 93 14:32:28 GMT Date: Mon, 4 Jan 93 14:32:28 GMT From: jim@meiko.co.uk (James Cownie) Message-Id: <9301041432.AA15874@hub.meiko.co.uk> Received: by float.co.uk (5.0/SMI-SVR4) id AA02690; Mon, 4 Jan 93 14:31:52 GMT To: mpi-pt2pt@cs.utk.edu, mpi-lang@cs.utk.edu Subject: Profilers etc. Content-Length: 2528 Gentlepeople, I have an implementation issue which I would like to raise in the MPI forum, since it is unclear where it fits into the sub-committee structure I have mailed to both mpi-pt2pt and mpi-lang. Apologies to those of you who receive this message twice. Issue ===== The major objective of MPI1 is to achieve portability of applications. This has major benefits for us all (not least in legitimising and therefore growing the MPP marketplace). One of the benefits which it would also be nice to achieve would be the wide availability of different tools which support programming in the MPI1 model. The most immediately obvious such tools (to me at least !) are 1) HPF to Fortran + MPI1 translators 2) Performance monitoring/tuning tools 3) Debuggers Support for the first is easy (since it just requires what we're already doing). Portable support for the second is not so trivial, since the collection of useful performance information is much more intrusive. Portable support for the third is harder still, and I won't discuss it further. Options ======= We have various possible options which we can take. 1) Ignore the problem Provide no support for portable performance monitoring tools, and leave each tool provider with a large porting problem. I don't like this solution, it loses some of the benefit of the standard, which should be attracting people to build tools. 2) Document specific implementation hooks as part of MPI1. In effect these would be callbacks from the library to profiling code which could then do whatever it liked. 3) As 2, but without REQUIRING that a conforming implementation provide the functions. They're there as a recommendation, rather than being mandatory. I think we should be concerned about this, and I'd like us at least to make some recommendation. Personally I'd probably implement two separate interface to the library, one of which provided the hooks, and the other of which didn't so that you don't pay the cost of checking the profiling hooks unless you asked to. Of course even if we do nothing it's not too hard to escape (using horrible macros) in C, but Fortran doesn't always have macros, so a properly specified internal solution is definitely preferable. Thoughts ??? Flame me at Dallas. I'm travelling tomorrow. When are we having a meeting in Europe ??? -- Jim James Cownie Meiko Limited 650 Aztec West Bristol BS12 4SD England Phone : +44 454 616171 FAX : +44 454 618188 E-Mail: jim@meiko.co.uk or jim@meiko.com From owner-mpi-pt2pt@CS.UTK.EDU Mon Jan 4 10:26:01 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA21722; Mon, 4 Jan 93 10:26:01 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA11937; Mon, 4 Jan 93 10:25:39 -0500 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Mon, 04 Jan 1993 15:25:38 GMT Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from msr.EPM.ORNL.GOV by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA11921; Mon, 4 Jan 93 10:25:36 -0500 Received: by msr.EPM.ORNL.GOV (5.61/1.34) id AA02891; Mon, 4 Jan 93 10:25:20 -0500 Date: Mon, 4 Jan 93 10:25:20 -0500 From: geist@msr.EPM.ORNL.GOV (Al Geist) Message-Id: <9301041525.AA02891@msr.EPM.ORNL.GOV> To: jim@meiko.co.uk, mpi-pt2pt@cs.utk.edu Subject: NRECV example explained. Hi Jim, Your points are well taken given your suggested additional goals for MPI. You asked some questions about an example you gave. char buffer[100]; int i; for (i=0; i<10; i++) { if (MPI_NRECV(&buffer[10*i],10,1,1,1)) break; } > 1) Into which elements of buffer will the receive occur ? The MPI_NRECV call says I want to receive a message not to exceed 10 bytes which matches type = 1, groupID = 1, source = 1. If the message has already arrived before the NRECV call, then it will be placed in buffer[0:9]. If it arrives while the 'for' loop is still executing, then it will be placed in buffer[10*i:10*i+10]. If it arrives later, then the user will need to ask for it later with either MPI_NRECV or MPI_RECV. All he can say for sure is the message did not arrive while I was polling for it. It is a strange example in that the user asks for a single message but specifies 10 different places where it should be stored. Ultimately, it the successful RECV or NRECV that specifies where the message is stored. > 2) Should I now do a blocking MPI_RECV as well ? Or does this only > make sense if the MPI_NRECVs all fail ? But if they do fail, is > the receive buffer still queued to receive a matching message when > it becomes available ? (and if so how do I cancel the outstanding > receive and get back ownership of my buffer ?) It only makes sense to call MPI_RECV if all MPI_NRECVs fail. It is not specified that the recv buffer ever has to be queued. An efficient immplementation may queue a failed NRECV in the educated guess that the user will ask for the message later in the same buffer. Then when the user does ask and the RECV or NRECV is successful the system avoids one copy of the message. Technically the user never loses ownership of the buffer. He can do anything he wants with it. If he is sure the posted message will never arrive, then he can post a message from another source to that same buffer. Cheers! Al From owner-mpi-pt2pt@CS.UTK.EDU Mon Jan 4 10:44:23 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA22293; Mon, 4 Jan 93 10:44:23 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA12552; Mon, 4 Jan 93 10:44:12 -0500 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Mon, 04 Jan 1993 15:44:11 GMT Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from marge.meiko.com by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA12544; Mon, 4 Jan 93 10:44:08 -0500 Received: from hub.meiko.co.uk by marge.meiko.com with SMTP id AA08401 (5.65c/IDA-1.4.4 for ); Mon, 4 Jan 1993 10:44:05 -0500 Received: from float.co.uk (float.meiko.co.uk) by hub.meiko.co.uk (4.1/SMI-4.1) id AA16038; Mon, 4 Jan 93 15:44:00 GMT Date: Mon, 4 Jan 93 15:44:00 GMT From: jim@meiko.co.uk (James Cownie) Message-Id: <9301041544.AA16038@hub.meiko.co.uk> Received: by float.co.uk (5.0/SMI-SVR4) id AA02725; Mon, 4 Jan 93 15:43:24 GMT To: geist@msr.EPM.ORNL.GOV Cc: mpi-pt2pt@cs.utk.edu In-Reply-To: Al Geist's message of Mon, 4 Jan 93 10:25:20 -0500 <9301041525.AA02891@msr.EPM.ORNL.GOV> Subject: NRECV example explained. Content-Length: 1350 Thanks Al, I now understand what you're trying to do, but I still don't think that your suggested "efficient implementation" is safe. Consider an example like this (less perverted than my previous one, which was of course constructed solely to make a point). strcpy(buffer0,"Something"); if (MPI_NRECV(buffer0, 10, 1, 1, 1)) /* Whatever */ else { sleep(20); /* Wait for the message to arrive, * For the sake of argument assume it does. */ MPI_RECV(buffer1, 10, 1, 1, 1); if (strcmp(buffer0,"Something") != SAME) bug("Buffer changed"); } Your suggested "efficient" implementation will surely overwrite buffer0 (which it should NOT do in your semantic model, since "Ultimately, it the successful RECV or NRECV that specifies where the message is stored."). I have to say I much prefer a model in which the queuing and dequeuing of buffers is explicit. Then it is clear where the current ownership of the buffer resides at any time, and when the additional copy can be avoided. (Though of course there is a lovely race to be resolved in the cancel...) Maybe we can talk about this further over a beer if you're going to be in Dallas. -- Jim James Cownie Meiko Limited 650 Aztec West Bristol BS12 4SD England Phone : +44 454 616171 FAX : +44 454 618188 E-Mail: jim@meiko.co.uk or jim@meiko.com From owner-mpi-pt2pt@CS.UTK.EDU Tue Jan 5 12:37:43 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA20368; Tue, 5 Jan 93 12:37:43 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA13588; Tue, 5 Jan 93 12:37:19 -0500 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Tue, 05 Jan 1993 17:37:18 GMT Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from fslg8.fsl.noaa.gov by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA13576; Tue, 5 Jan 93 12:37:16 -0500 Received: by fslg8.fsl.noaa.gov (5.57/Ultrix3.0-C) id AA22236; Tue, 5 Jan 93 17:37:12 GMT Received: by macaw.fsl.noaa.gov (4.1/SMI-4.1) id AA00742; Tue, 5 Jan 93 10:36:12 MST Date: Tue, 5 Jan 93 10:36:12 MST From: hender@macaw.fsl.noaa.gov (Tom Henderson) Message-Id: <9301051736.AA00742@macaw.fsl.noaa.gov> To: mpi-pt2pt@cs.utk.edu Subject: Re: Multifarious I have a few short comments on Jim's comments... > > Removing internal state > ======================= > > I'd also like to remove the hidden state ("last message"), there are > various ways of achieving this, none of which is exceptionally > attractive. > > 1) Return a structure from MPI_RECV (and friends) > This is fine in C, but not good in Fortran 77 (and may be more > expensive than we want when you don't need any of the information). > > e.g. > typedef struct MPI_REPLY > { > int status; > int nbytes; > int whoFrom; > int tag; > /* More ? */ > } MPI_REPLY; > > { > MPI_REPLY res; > > res = MPI_RECV(...); > > if (res.whoFrom == 0) > > ... etc ... > } > > > 2) Return results indirect through individual parameters (arguments for the > Fortran inclined) > > This works, but is very error prone, particularly in Fortran when > the arguments are INOUT. (We have an interface like this at the > moment, and it does cause problems). > > 3) Return a tag on which queries can be made. > > This means that you must then have another call to the system to > dispose of the tag (and associated system storage). This is > unpleasant on a simple blocking receive when you don't want any of > the information (since it's another call to the library). > > 4) Have additional (possibly optional) OUT arguments through which the > results are returned if required. > > Using optional keyword arguments in F90 this works fine. > C can be made to work more or less. > F77 can't (within the standard). > > 5) Have one required argument which is a pointer to a reply structure > like that used in 1) which is then filled in by the function. > > This can work in all the languages. > In C you could also allow NULL to mean no such results are > required. > > > IMHO Solution 5 is best. > > If we wanted to (and we may) then we could disallow direct access to > this structure and provide functions to pick it apart (these could be > macros in C, inline functions in C++). This would avoid having to > specify the contents of the structure and allow for future expansion. > > > > -- Jim > James Cownie > Meiko Limited > 650 Aztec West > Bristol BS12 4SD > England I also like Solution 5 and like the idea of providing functions to access the reply structure. "Hiding" the data structure in this way makes it easy to change things in later revisions when we (inevitably) find out we've forgotten something. Tom Henderson Forecast Systems Laboratory (NOAA) hender@fsl.noaa.gov From owner-mpi-pt2pt@CS.UTK.EDU Tue Jan 5 15:52:33 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA01504; Tue, 5 Jan 93 15:52:33 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA22365; Tue, 5 Jan 93 15:51:40 -0500 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Tue, 05 Jan 1993 20:51:34 GMT Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from antares.mcs.anl.gov by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA22325; Tue, 5 Jan 93 15:50:59 -0500 Received: from godzilla.mcs.anl.gov by antares.mcs.anl.gov (4.1/SMI-GAR) id AA21624; Tue, 5 Jan 93 14:50:42 CST From: gropp@antares.mcs.anl.gov (William Gropp) Received: by godzilla.mcs.anl.gov (4.1/GeneV4) id AA17798; Tue, 5 Jan 93 14:50:40 CST Date: Tue, 5 Jan 93 14:50:40 CST Message-Id: <9301052050.AA17798@godzilla.mcs.anl.gov> To: mpi-pt2pt@cs.utk.edu Subject: Comments on "ready receiver" part of draft implementation Here is some text supporting the idea of "ready receivers". Bill Gropp and Rusty Lusk %!PS-Adobe-2.0 %%Creator: dvips, version 5.4 (C) 1986-90 Radical Eye Software %%Title: rr.dvi %%Pages: 3 1 %%BoundingBox: 0 0 612 792 %%EndComments %%BeginProcSet: tex.pro /TeXDict 200 dict def TeXDict begin /N /def load def /B{bind def}N /S /exch load def /X{S N}B /TR /translate load N /isls false N /vsize 10 N /@rigin{ isls{[0 1 -1 0 0 0]concat}if 72 Resolution div 72 VResolution div neg scale Resolution VResolution vsize neg mul TR}B /@letter{/vsize 10 N}B /@landscape{ /isls true N /vsize -1 N}B /@a4{/vsize 10.6929133858 N}B /@a3{/vsize 15.5531 N }B /@ledger{/vsize 16 N}B /@legal{/vsize 13 N}B /@manualfeed{statusdict /manualfeed true put}B /@copies{/#copies X}B /FMat[1 0 0 -1 0 0]N /FBB[0 0 0 0 ]N /df{/sf 1 N /fntrx FMat N df-tail}B /dfs{div /sf X /fntrx[sf 0 0 sf neg 0 0 ]N df-tail}B /df-tail{/nn 8 dict N nn begin /FontType 3 N /FontMatrix fntrx N /FontBBox FBB N string /base X array /BitMaps X /BuildChar{CharBuilder}N /Encoding IE N end dup{/foo setfont}2 array copy cvx N load 0 nn put /ctr 0 N[ }B /E{pop nn dup definefont setfont}B /ch-image{ch-data dup type /stringtype ne{ctr get /ctr ctr 1 add N}if}B /ch-width{ch-data dup length 5 sub get}B /ch-height{ch-data dup length 4 sub get}B /ch-xoff{128 ch-data dup length 3 sub get sub}B /ch-yoff{ch-data dup length 2 sub get 127 sub}B /ch-dx{ch-data dup length 1 sub get}B /ctr 0 N /CharBuilder{save 3 1 roll S dup /base get 2 index get S /BitMaps get S get /ch-data X pop /ctr 0 N ch-dx 0 ch-xoff ch-yoff ch-height sub ch-xoff ch-width add ch-yoff setcachedevice ch-width ch-height true[1 0 0 -1 -.1 ch-xoff sub ch-yoff .1 add]{ch-image}imagemask restore}B /D{ /cc X dup type /stringtype ne{]}if nn /base get cc ctr put nn /BitMaps get S ctr S sf 1 ne{dup dup length 1 sub dup 2 index S get sf div put}if put /ctr ctr 1 add N}B /I{cc 1 add D}B /bop{userdict /bop-hook known{bop-hook}if /SI save N @rigin 0 0 moveto}B /eop{clear SI restore showpage userdict /eop-hook known{eop-hook}if}B /@start{userdict /start-hook known{start-hook}if /VResolution X /Resolution X 1000 div /DVImag X /IE 256 array N 0 1 255{IE S 1 string dup 0 3 index put cvn put}for}B /p /show load N /RMat[1 0 0 -1 0 0]N /BDot 8 string N /v{/ruley X /rulex X V}B /V{gsave TR -.1 -.1 TR rulex ruley scale 1 1 false RMat{BDot}imagemask grestore}B /a{moveto}B /delta 0 N /tail{ dup /delta X 0 rmoveto}B /M{S p delta add tail}B /b{S p tail}B /c{-4 M}B /d{ -3 M}B /e{-2 M}B /f{-1 M}B /g{0 M}B /h{1 M}B /i{2 M}B /j{3 M}B /k{4 M}B /l{p -4 w}B /m{p -3 w}B /n{p -2 w}B /o{p -1 w}B /q{p 1 w}B /r{p 2 w}B /s{p 3 w}B /t {p 4 w}B /w{0 rmoveto}B /x{0 S rmoveto}B /y{3 2 roll p a}B /bos{/SS save N}B /eos{clear SS restore}B end %%EndProcSet %%BeginProcSet: special.pro TeXDict begin /SDict 200 dict N SDict begin /@SpecialDefaults{/hs 612 N /vs 792 N /ho 0 N /vo 0 N /hsc 1 N /vsc 1 N /ang 0 N /CLIP false N /BBcalc false N /p 3 def}B /@scaleunit 100 N /@hscale{@scaleunit div /hsc X}B /@vscale{ @scaleunit div /vsc X}B /@hsize{/hs X /CLIP true N}B /@vsize{/vs X /CLIP true N}B /@hoffset{/ho X}B /@voffset{/vo X}B /@angle{/ang X}B /@rwi{10 div /rwi X} B /@llx{/llx X}B /@lly{/lly X}B /@urx{/urx X}B /@ury{/ury X /BBcalc true N}B /magscale true def end /@MacSetUp{userdict /md known{userdict /md get type /dicttype eq{md begin /letter{}N /note{}N /legal{}N /od{txpose 1 0 mtx defaultmatrix dtransform S atan/pa X newpath clippath mark{transform{ itransform moveto}}{transform{itransform lineto}}{6 -2 roll transform 6 -2 roll transform 6 -2 roll transform{itransform 6 2 roll itransform 6 2 roll itransform 6 2 roll curveto}}{{closepath}}pathforall newpath counttomark array astore /gc xdf pop ct 39 0 put 10 fz 0 fs 2 F/|______Courier fnt invertflag{ PaintBlack}if}N /txpose{pxs pys scale ppr aload pop por{noflips{pop S neg S TR pop 1 -1 scale}if xflip yflip and{pop S neg S TR 180 rotate 1 -1 scale ppr 3 get ppr 1 get neg sub neg ppr 2 get ppr 0 get neg sub neg TR}if xflip yflip not and{pop S neg S TR pop 180 rotate ppr 3 get ppr 1 get neg sub neg 0 TR}if yflip xflip not and{ppr 1 get neg ppr 0 get neg TR}if}{noflips{TR pop pop 270 rotate 1 -1 scale}if xflip yflip and{TR pop pop 90 rotate 1 -1 scale ppr 3 get ppr 1 get neg sub neg ppr 2 get ppr 0 get neg sub neg TR}if xflip yflip not and{TR pop pop 90 rotate ppr 3 get ppr 1 get neg sub neg 0 TR}if yflip xflip not and{TR pop pop 270 rotate ppr 2 get ppr 0 get neg sub neg 0 S TR}if} ifelse scaleby96{ppr aload pop 4 -1 roll add 2 div 3 1 roll add 2 div 2 copy TR .96 dup scale neg S neg S TR}if}N /cp{pop pop showpage pm restore}N end}if} if}N /normalscale{Resolution 72 div VResolution 72 div neg scale magscale{ DVImag dup scale}if}N /psfts{S 65536 div N}N /startTexFig{/psf$SavedState save N userdict maxlength dict begin /magscale false def normalscale currentpoint TR /psf$ury psfts /psf$urx psfts /psf$lly psfts /psf$llx psfts /psf$y psfts /psf$x psfts currentpoint /psf$cy X /psf$cx X /psf$sx psf$x psf$urx psf$llx sub div N /psf$sy psf$y psf$ury psf$lly sub div N psf$sx psf$sy scale psf$cx psf$sx div psf$llx sub psf$cy psf$sy div psf$ury sub TR /showpage{}N /erasepage{}N /copypage{}N @MacSetUp}N /doclip{psf$llx psf$lly psf$urx psf$ury currentpoint 6 2 roll newpath 4 copy 4 2 roll moveto 6 -1 roll S lineto S lineto S lineto closepath clip newpath moveto}N /endTexFig{end psf$SavedState restore}N /@beginspecial{SDict begin /SpecialSave save N gsave normalscale currentpoint TR @SpecialDefaults}B /@setspecial{CLIP{newpath 0 0 moveto hs 0 rlineto 0 vs rlineto hs neg 0 rlineto closepath clip}{initclip}ifelse ho vo TR hsc vsc scale ang rotate BBcalc{rwi urx llx sub div dup scale llx neg lly neg TR}if /showpage{}N /erasepage{}N /copypage{}N newpath}B /@endspecial{grestore clear SpecialSave restore end}B /@defspecial{SDict begin}B /@fedspecial{end}B /li{lineto}B /rl{rlineto}B /rc{rcurveto}B /np{/SaveX currentpoint /SaveY X N 1 setlinecap newpath}B /st{stroke SaveX SaveY moveto}B /fil{fill SaveX SaveY moveto}B /ellipse{/endangle X /startangle X /yrad X /xrad X /savematrix matrix currentmatrix N TR xrad yrad scale 0 0 1 startangle endangle arc savematrix setmatrix}B end %%EndProcSet TeXDict begin 1000 300 300 @start /Fa 58 123 df<001F83E000F06E3001C078780380F8 780300F03007007000070070000700700007007000070070000700700007007000FFFFFF800700 700007007000070070000700700007007000070070000700700007007000070070000700700007 007000070070000700700007007000070070000700700007007000070070007FE3FF001D20809F 1B>11 D<003F0000E0C001C0C00381E00701E00701E00700000700000700000700000700000700 00FFFFE00700E00700E00700E00700E00700E00700E00700E00700E00700E00700E00700E00700 E00700E00700E00700E00700E00700E00700E07FC3FE1720809F19>I<7038F87CFC7EFC7E743A 0402040204020804080410081008201040200F0E7E9F17>34 D<70F8FCFC740404040808101020 40060E7C9F0D>39 D<0020004000800100020006000C000C001800180030003000300070006000 60006000E000E000E000E000E000E000E000E000E000E000E000E0006000600060007000300030 003000180018000C000C000600020001000080004000200B2E7DA112>I<800040002000100008 000C00060006000300030001800180018001C000C000C000C000E000E000E000E000E000E000E0 00E000E000E000E000E000C000C000C001C001800180018003000300060006000C000800100020 00400080000B2E7DA112>I<70F8FCFC74040404080810102040060E7C840D>44 DI<70F8F8F87005057C840D>I<000100030003000600060006000C000C 000C00180018001800300030003000600060006000C000C000C001800180018003000300030006 00060006000C000C000C00180018001800300030003000600060006000C000C000C000102D7DA1 17>I<03F0000E1C001C0E00180600380700700380700380700380700380F003C0F003C0F003C0 F003C0F003C0F003C0F003C0F003C0F003C0F003C0F003C0F003C0F003C0700380700380700380 7807803807001806001C0E000E1C0003F000121F7E9D17>I<018003800F80F380038003800380 038003800380038003800380038003800380038003800380038003800380038003800380038003 80038007C0FFFE0F1E7C9D17>I<03F0000C1C00100E00200700400780800780F007C0F803C0F8 03C0F803C02007C00007C0000780000780000F00000E00001C0000380000700000600000C00001 80000300000600400C00401800401000803FFF807FFF80FFFF80121E7E9D17>I<03F0000C1C00 100E00200F00780F80780780780780380F80000F80000F00000F00000E00001C0000380003F000 003C00000E00000F000007800007800007C02007C0F807C0F807C0F807C0F00780400780400F00 200E001C3C0003F000121F7E9D17>I<007C000182000701000E03800C07801C07803803003800 00780000700000700000F1F000F21C00F40600F80700F80380F80380F003C0F003C0F003C0F003 C0F003C07003C07003C07003803803803807001807000C0E00061C0001F000121F7E9D17>54 D<03F0000C0C001006003003002001806001806001806001807001807803003E03003F06001FC8 000FF00003F80007FC000C7E00103F00300F806003804001C0C001C0C000C0C000C0C000C0C000 806001802001001002000C0C0003F000121F7E9D17>56 D<70F8F8F87000000000000000000000 70F8F8F87005147C930D>58 D<000100000003800000038000000380000007C0000007C0000007 C0000009E0000009E0000009E0000010F0000010F0000010F00000207800002078000020780000 403C0000403C0000403C0000801E0000801E0000FFFE0001000F0001000F0001000F0002000780 0200078002000780040003C00E0003C01F0007E0FFC03FFE1F207F9F22>65 D<000FC040007030C001C009C0038005C0070003C00E0001C01E0000C01C0000C03C0000C07C00 00407C00004078000040F8000000F8000000F8000000F8000000F8000000F8000000F8000000F8 000000F8000000780000007C0000407C0000403C0000401C0000401E0000800E00008007000100 0380020001C0040000703800000FC0001A217D9F21>67 DI70 D73 D76 DII<001F800000F0F00001C0380007801E000F000F000E0007001E0007803C 0003C03C0003C07C0003E0780001E0780001E0F80001F0F80001F0F80001F0F80001F0F80001F0 F80001F0F80001F0F80001F0F80001F0780001E07C0003E07C0003E03C0003C03C0003C01E0007 800E0007000F000F0007801E0001C0380000F0F000001F80001C217D9F23>II82 D<07E0800C198010078030038060 0180600180E00180E00080E00080E00080F00000F000007800007F00003FF0001FFC000FFE0003 FF00001F800007800003C00003C00001C08001C08001C08001C08001C0C00180C00380E00300F0 0600CE0C0081F80012217D9F19>I<7FFFFFE0780F01E0600F0060400F0020400F0020C00F0030 800F0010800F0010800F0010800F0010000F0000000F0000000F0000000F0000000F0000000F00 00000F0000000F0000000F0000000F0000000F0000000F0000000F0000000F0000000F0000000F 0000000F0000000F0000000F0000001F800007FFFE001C1F7E9E21>I87 D<080410082010201040204020804080408040B85CFC7EFC7E7C3E381C0F0E7B9F17>92 D<1FE000303000781800781C00300E00000E00000E00000E0000FE00078E001E0E00380E00780E 00F00E10F00E10F00E10F01E10781E103867200F83C014147E9317>97 D<0E0000FE00000E0000 0E00000E00000E00000E00000E00000E00000E00000E00000E00000E3E000EC3800F01C00F00E0 0E00E00E00700E00700E00780E00780E00780E00780E00780E00780E00700E00700E00E00F00E0 0D01C00CC300083E0015207F9F19>I<03F80E0C1C1E381E380C70007000F000F000F000F000F0 00F00070007000380138011C020E0C03F010147E9314>I<000380003F80000380000380000380 00038000038000038000038000038000038000038003E380061B801C0780380380380380700380 700380F00380F00380F00380F00380F00380F003807003807003803803803807801C07800E1B80 03E3F815207E9F19>I<03F0000E1C001C0E00380700380700700700700380F00380F00380FFFF 80F00000F00000F000007000007000003800801800800C010007060001F80011147F9314>I<00 7C00C6018F038F07060700070007000700070007000700FFF00700070007000700070007000700 070007000700070007000700070007000700070007007FF01020809F0E>I<0000E003E3300E3C 301C1C30380E00780F00780F00780F00780F00780F00380E001C1C001E380033E0002000002000 003000003000003FFE001FFF800FFFC03001E0600070C00030C00030C00030C000306000603000 C01C038003FC00141F7F9417>I<0E0000FE00000E00000E00000E00000E00000E00000E00000E 00000E00000E00000E00000E3E000E43000E81800F01C00F01C00E01C00E01C00E01C00E01C00E 01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C0FFE7FC16207F9F19>I< 1C001E003E001E001C000000000000000000000000000E007E000E000E000E000E000E000E000E 000E000E000E000E000E000E000E000E000E000E00FFC00A1F809E0C>I<00E001F001F001F000 E0000000000000000000000000007007F000F00070007000700070007000700070007000700070 007000700070007000700070007000700070007000706070F060F0C061803F000C28829E0E>I< 0E0000FE00000E00000E00000E00000E00000E00000E00000E00000E00000E00000E00000E0FF0 0E03C00E03000E02000E04000E08000E10000E30000E70000EF8000F38000E1C000E1E000E0E00 0E07000E07800E03800E03C00E03E0FFCFF815207F9F18>I<0E00FE000E000E000E000E000E00 0E000E000E000E000E000E000E000E000E000E000E000E000E000E000E000E000E000E000E000E 000E000E000E000E00FFE00B20809F0C>I<0E1F01F000FE618618000E81C81C000F00F00E000F 00F00E000E00E00E000E00E00E000E00E00E000E00E00E000E00E00E000E00E00E000E00E00E00 0E00E00E000E00E00E000E00E00E000E00E00E000E00E00E000E00E00E000E00E00E00FFE7FE7F E023147F9326>I<0E3E00FE43000E81800F01C00F01C00E01C00E01C00E01C00E01C00E01C00E 01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C0FFE7FC16147F9319>I<01F800 070E001C03803801C03801C07000E07000E0F000F0F000F0F000F0F000F0F000F0F000F07000E0 7000E03801C03801C01C0380070E0001F80014147F9317>I<0E3E00FEC3800F01C00F00E00E00 E00E00F00E00700E00780E00780E00780E00780E00780E00780E00700E00F00E00E00F01E00F01 C00EC3000E3E000E00000E00000E00000E00000E00000E00000E00000E0000FFE000151D7F9319 >I<03E0800619801C05803C0780380380780380700380F00380F00380F00380F00380F00380F0 03807003807803803803803807801C0B800E138003E38000038000038000038000038000038000 0380000380000380003FF8151D7E9318>I<0E78FE8C0F1E0F1E0F0C0E000E000E000E000E000E 000E000E000E000E000E000E000E000E00FFE00F147F9312>I<1F9030704030C010C010C010E0 0078007F803FE00FF00070803880188018C018C018E030D0608F800D147E9312>I<0200020002 00060006000E000E003E00FFF80E000E000E000E000E000E000E000E000E000E000E000E080E08 0E080E080E080610031001E00D1C7F9B12>I<0E01C0FE1FC00E01C00E01C00E01C00E01C00E01 C00E01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C00E03C00603C0030DC001F1 FC16147F9319>III<7FC3FC0F01E00701C007018003810001C20000E40000EC00007800003800003C00 007C00004E000087000107000303800201C00601E01E01E0FF07FE1714809318>II<3FFF380E200E201C40384078407000E001E001C00380078007010E01 1E011C0338027006700EFFFE10147F9314>I E /Fb 14 122 df<01FFFFC0001E00F0001E0078 001E0038001E003C003C003C003C003C003C003C003C003C0078007800780078007800F0007801 E000F0078000FFFE0000F00F8000F003C001E001C001E001E001E001E001E001E003C001E003C0 01E003C001E003C001C0078003C00780078007800F0007801E000F007800FFFFE0001E1F7D9E20 >66 D<0000FC040007030C001C00980030007800E0007801C00038038000300380003007000030 0E0000301E0000201E0000203C0000003C00000078000000780000007800000078000000F00000 00F000FFF0F0000780F0000780F0000F0070000F0070000F0070000F0070001E0038001E001800 3E001C002E000E00CC000383040000FC00001E217A9F23>71 D<01FFF800001F0000001E000000 1E0000001E0000003C0000003C0000003C0000003C000000780000007800000078000000780000 00F0000000F0000000F0000000F0000001E0000001E0000001E0000001E0008003C0010003C001 0003C0030003C00200078006000780060007800C0007801C000F007800FFFFF800191F7D9E1D> 76 D<01FFFF00001E03C0001E00E0001E0070001E0078003C0078003C0078003C0078003C0078 007800F0007800F0007801E0007801C000F0070000F01E0000FFF00000F0380001E01C0001E01E 0001E00E0001E00F0003C01E0003C01E0003C01E0003C01E0007803C0007803C0807803C080780 3C100F801C10FFF00C20000007C01D207D9E21>82 D<00C001E001E001C0000000000000000000 00000000000E003300230043804300470087000E000E000E001C001C001C003840388030807080 310033001C000B1F7C9E0E>105 D<01E0000FE00001C00001C00001C00001C000038000038000 0380000380000700000700000703C00704200E08E00E11E00E21E00E40C01C80001D00001E0000 1FC00038E000387000387000383840707080707080707080703100E03100601E0013207D9F15> 107 D<03C01FC0038003800380038007000700070007000E000E000E000E001C001C001C001C00 38003800380038007000700070007100E200E200E200E200640038000A207C9F0C>I<007C0001 C3000301800E01C01E01C01C01E03C01E07801E07801E07801E0F003C0F003C0F003C0F00780F0 0700700F00700E0030180018700007C00013147C9317>111 D<01C1E002621804741C04781C04 701E04701E08E01E00E01E00E01E00E01E01C03C01C03C01C03C01C0380380780380700380E003 C1C0072380071E000700000700000E00000E00000E00000E00001C00001C0000FFC000171D8093 17>I<1C1E002661004783804787804707804703008E00000E00000E00000E00001C00001C0000 1C00001C000038000038000038000038000070000030000011147C9313>114 D<00FC030206010C030C070C060C000F800FF007F803FC003E000E700EF00CF00CE00840102060 1F8010147D9313>I<018001C0038003800380038007000700FFF007000E000E000E000E001C00 1C001C001C003800380038003820704070407080708031001E000C1C7C9B0F>I<0E00C03300E0 2301C04381C04301C04701C08703800E03800E03800E03801C07001C07001C07001C07101C0E20 180E20180E201C1E200C264007C38014147C9318>I<0E00C03300E02301C04381C04301C04701 C08703800E03800E03800E03801C07001C07001C07001C07001C0E00180E00180E001C1E000C3C 0007DC00001C00001C00003800F03800F07000E06000C0C0004380003E0000131D7C9316>121 D E /Fc 20 117 df45 D<000003800000000007C00000000007C0000000000FE0000000000FE0000000000FE000000000 1FF0000000001FF0000000003FF8000000003FF8000000003FF80000000073FC0000000073FC00 000000F3FE00000000E1FE00000000E1FE00000001C0FF00000001C0FF00000003C0FF80000003 807F80000007807FC0000007003FC0000007003FC000000E003FE000000E001FE000001E001FF0 00001C000FF000001FFFFFF000003FFFFFF800003FFFFFF80000780007FC0000700003FC000070 0003FC0000E00001FE0000E00001FE0001E00001FF0001C00000FF0001C00000FF00FFFE001FFF FEFFFE001FFFFEFFFE001FFFFE2F297EA834>65 D 72 D77 DI80 D<01FF800007FFF0000F81F8001FC0 7E001FC07E001FC03F000F803F8007003F8000003F8000003F8000003F80000FFF8000FFFF8007 FC3F800FE03F803F803F803F003F807F003F80FE003F80FE003F80FE003F80FE003F807E007F80 7F00DF803F839FFC0FFF0FFC01FC03FC1E1B7E9A21>97 D<001FF80000FFFE0003F01F0007E03F 800FC03F801F803F803F801F007F800E007F0000007F000000FF000000FF000000FF000000FF00 0000FF000000FF000000FF0000007F0000007F0000007F8000003F8001C01F8001C00FC0038007 E0070003F01E0000FFFC00001FE0001A1B7E9A1F>99 D<003FE00001FFF80003F07E0007C01F00 0F801F801F800F803F800FC07F000FC07F0007C07F0007E0FF0007E0FF0007E0FFFFFFE0FFFFFF E0FF000000FF000000FF0000007F0000007F0000007F0000003F8000E01F8000E00FC001C007E0 038003F81F0000FFFE00001FF0001B1B7E9A20>101 D<0007F0003FFC00FE3E01F87F03F87F03 F07F07F07F07F03E07F00007F00007F00007F00007F00007F00007F000FFFFC0FFFFC0FFFFC007 F00007F00007F00007F00007F00007F00007F00007F00007F00007F00007F00007F00007F00007 F00007F00007F00007F00007F00007F00007F00007F0007FFF807FFF807FFF80182A7EA915>I< 00FF81F003FFE7F80FC1FE7C1F80FC7C1F007C383F007E107F007F007F007F007F007F007F007F 007F007F007F007F003F007E001F007C001F80FC000FC1F8001FFFE00018FF8000380000003800 00003C0000003E0000003FFFF8001FFFFF001FFFFF800FFFFFC007FFFFE01FFFFFF03E0007F07C 0001F8F80000F8F80000F8F80000F8F80000F87C0001F03C0001E01F0007C00FC01F8003FFFE00 007FF0001E287E9A22>II<07000F801FC0 3FE03FE03FE01FC00F8007000000000000000000000000000000FFE0FFE0FFE00FE00FE00FE00F E00FE00FE00FE00FE00FE00FE00FE00FE00FE00FE00FE00FE00FE00FE00FE00FE00FE0FFFEFFFE FFFE0F2B7DAA14>I108 DII<003FE00001FFFC0003F07E000FC01F801F800FC03F800FE03F0007E0 7F0007F07F0007F07F0007F0FF0007F8FF0007F8FF0007F8FF0007F8FF0007F8FF0007F8FF0007 F8FF0007F87F0007F07F0007F03F800FE03F800FE01F800FC00FC01F8007F07F0001FFFC00003F E0001D1B7E9A22>I114 D<03FE300FFFF01E03F03800F0700070F00070 F00070F80070FC0000FFE0007FFE007FFF803FFFE01FFFF007FFF800FFF80003FC0000FC60007C E0003CF0003CF00038F80038FC0070FF01E0F7FFC0C1FF00161B7E9A1B>I<0070000070000070 0000700000F00000F00000F00001F00003F00003F00007F0001FFFF0FFFFF0FFFFF007F00007F0 0007F00007F00007F00007F00007F00007F00007F00007F00007F00007F00007F00007F03807F0 3807F03807F03807F03807F03803F03803F87001F86000FFC0001F8015267FA51B>I E end %%EndProlog %%BeginSetup %%Feature: *Resolution 300 TeXDict begin @letter /letter where {pop letter} if %%EndSetup %%Page: 1 1 bop 117 7 a Fc(A)23 b(Note)f(on)i(Access)e(to)h(High-P)n(erformance)e (Message-P)n(assing)832 94 y(Proto)r(cols)869 240 y Fb(Bil)r(l)16 b(Gr)n(opp)864 297 y(R)o(usty)g(Lusk)526 353 y Fa(Mathematics)f(and)g (Computer)g(Science)i(Division)678 409 y(Argonne)e(National)h(Lab)q(oratory) 145 546 y(The)22 b(MPI)h(Implemen)o(tation)g(T)l(estb)q(ed)g(con)o(tains)g (what)f(are)g(called)i(there)e(\\Ready-Receiv)o(er")74 602 y(v)o(ersions)17 b(of)g(the)g(message-passing)g(primitiv)o(es.)27 b(The)17 b(purp)q(ose)h(of)e(this)i(brief)f(note)g(is)h(to)e(giv)o(e)h(some) 74 659 y(explanation)e(of)f(what)g(wh)o(y)f(w)o(e)h(think)h(something)g(lik)o (e)g(this)f(needs)h(to)f(b)q(e)h(considered)g(as)f(part)f(of)h(the)74 715 y(ev)o(en)o(tual)i(standard)e(and)i(to)e(rep)q(ort)h(on)g(some)g(exp)q (erimen)o(ts)h(that)f(supp)q(ort)g(this)h(opinion.)145 802 y(It)21 b(is)g(quite)h(p)q(ossible)h(to)e(enco)q(de)h(message-passing)f (algorithms)g(with)h(only)f(a)g(single)h(t)o(yp)q(e)g(of)74 858 y(send)d(and)g(receiv)o(e,)h(sa)o(y)e(a)g(blo)q(c)o(king)i(send)f(and)g (a)f(blo)q(c)o(king)i(receiv)o(e.)30 b(Most)18 b(of)g(the)g(v)m(ariations)h (are)74 914 y(to)c(enable)h(increased)g(p)q(erformance,)f(p)q(ossibly)i(at)d (the)h(exp)q(ense)i(of)d(increased)i(program)e(complexit)o(y)l(.)74 971 y(An)19 b(example)h(of)f(this)g(is)g(the)g(general)h(consensus)f(that)g (w)o(e)f(need)i(non-blo)q(c)o(king)h(send)e(and)g(receiv)o(e)74 1027 y(op)q(erations)c(in)i(order)d(to)h(o)o(v)o(erlap)g(computation)g(with)h (comm)o(unication.)145 1114 y(It)11 b(is)h(imp)q(ortan)o(t)f(not)g(to)g(stop) g(to)q(o)g(so)q(on,)g(ho)o(w)o(ev)o(er,)g(in)h(pro)o(viding)h(supp)q(ort)e (for)g(high)h(p)q(erformance.)74 1170 y(As)k(time)g(go)q(es)f(b)o(y)l(,)h(v)o (endors)f(will)j(pro)o(vide)e(op)q(erations)g(that)f(are)g(faster)g(than)g (the)h(ones)g(that)f(can)g(b)q(e)74 1227 y(sp)q(eci\014ed)i(with)d(the)g(op)q (erations)h(in)g(the)f(initial)j(standard)c(prop)q(osal.)20 b(It)15 b(is)f(p)q(ossible)i(to)e(an)o(ticipate)h(at)74 1283 y(least)g(some)g(of)g(these)h(op)q(erations)f(and)g(pro)o(vide)h(for)f(them)g (in)h(the)f(MPI)g(standard.)145 1370 y(One)j(example)g(o)q(ccurs)f(in)h(the)g (situation)f(when)h(receiv)o(es)g(ha)o(v)o(e)f(b)q(een)h(issued)g(prior)g(to) e(the)i(send)74 1426 y(op)q(eration)e(that)f(is)i(exp)q(ected)g(to)e(satisfy) h(them.)22 b(In)16 b(suc)o(h)g(a)g(case)g(a)f(considerable)j(amoun)o(t)d(of)h (hand-)74 1483 y(shaking)23 b(can)f(b)q(e)g(eliminated)i(since)f(the)f (bu\013ers)g(ha)o(v)o(e)g(already)g(b)q(een)h(allo)q(cated.)41 b(On)22 b(the)g(In)o(tel)74 1539 y(IPSC/860)c(and)g(the)g(Delta,)g(one)h(can) f(tak)o(e)f(adv)m(an)o(tage)h(of)g(this)g(b)o(y)g(using)h(\\force)f(t)o(yp)q (es".)28 b(Almost)74 1595 y(all)20 b(users)f(of)f(the)h(Delta)f(who)h(are)f (seeking)i(go)q(o)q(d)e(p)q(erformance)h(use)g(this)g(tec)o(hnique.)32 b(Its)19 b(name)g(is)74 1652 y(misleading,)d(since)f(it)f(do)q(esn't)g(ha)o (v)o(e)g(an)o(ything)g(to)f(do)h(with)h(t)o(yp)q(es.)k(The)14 b(imp)q(ortan)o(t)g(thing)g(to)g(realize)74 1708 y(is)19 b(that)f(it)i(is)f (not)f(just)h(an)f(arcane)h(v)o(endor-sp)q(eci\014c)i(optimization,)f(but)f (a)f(general)i(situation)f(that)74 1765 y(transcends)13 b(an)o(y)g (particular)h(implemen)o(tation.)20 b(The)13 b(general)h(situation)f(is)h (that)e(when)i(receiv)o(es)g(ha)o(v)o(e)74 1821 y(b)q(een)i(issued)h(ahead)e (of)g(time,)g(message)g(latency)h(can)f(b)q(e)h(greatly)f(decreased.)145 1908 y(W)l(e)e(conducted)h(some)f(preliminary)h(exp)q(erimen)o(ts)g(on)f(the) g(IPSC/860)g(with)g(the)g(MPI)g(implemen-)74 1964 y(tation)19 b(testb)q(ed,)g(using)h(the)f(\\rr")f(v)o(ersions)h(of)f(send)i(and)f(receiv) o(e)h(that)e(it)h(supplies.)33 b(\(\\rr")17 b(stands)74 2021 y(for)d(\\ready)g(receiv)o(er".\))20 b(The)15 b(test)f(program)f(just)i (ping-p)q(ongs)g(a)g(message)f(bac)o(k)g(and)h(forth)f(b)q(et)o(w)o(een)74 2077 y(t)o(w)o(o)20 b(no)q(des.)39 b(The)22 b(follo)o(wing)g(graphs)f(sho)o (w)g(the)h(p)q(erformance)f(impro)o(v)o(emen)o(ts)g(pro)o(vided)i(b)o(y)e (the)74 2134 y(ready-receiv)o(er)16 b(op)q(erations.)145 2220 y(This)g(next)f(test)f(sho)o(ws)h(the)g(b)q(eha)o(vior)h(in)g(sending)g (around)g(a)e(ring.)963 2790 y(1)p eop %%Page: 2 2 bop 74 2161 a @beginspecial 72 @llx 72 @lly 504 @urx 504 @ury 4320 @rwi @setspecial %%BeginDocument: ftime.ps .24 .24 scale /g0dict 40 dict def g0dict begin /m { moveto } bind def /a { rmoveto } bind def /l { lineto } bind def /v { 0 exch rlineto } bind def /h { 0 rlineto } bind def /s { stroke } bind def /n { newpath } bind def /r { rlineto } bind def /c { closepath } bind def /f { fill } bind def /p { copypage erasepage } def /g { setgray } bind def /b { newpath moveto lineto stroke } bind def /d { newpath moveto 0 exch rlineto stroke } bind def /e { newpath moveto 0 rlineto stroke } bind def /i { newpath moveto rlineto stroke } bind def /rs { dup stringwidth pop 0 exch sub 0 a show } bind def /cs { dup stringwidth pop 2 div 0 exch sub 0 a show } bind def /rshow { gsave currentpoint translate rotate 0 0 moveto show grestore } bind def /rrs { gsave currentpoint translate rotate 0 0 moveto dup stringwidth pop 0 exch sub 0 a show grestore } bind def /rcs { gsave currentpoint translate rotate 0 0 moveto dup stringwidth pop 2 div 0 exch sub 0 a show grestore } bind def 1440 569 569 e 36 569 569 d 18 710 569 d 36 850 569 d 18 991 569 d 36 1132 569 d 18 1272 569 d 36 1413 569 d 18 1553 569 d 36 1694 569 d 18 1834 569 d 36 1975 569 d n 568 558 m -4 -2 r -2 -3 r 560 546 l -4 v 562 536 l 2 -4 r 4 -1 r 3 h 3 1 r 3 4 r 1 6 r 4 v -1 7 r -3 3 r -3 2 r c s n 568 558 m -3 -2 r -1 -1 r -1 -2 r 562 546 l -4 v 563 536 l 1 -3 r 1 -1 r 3 -1 r s n 571 531 m 2 1 r 1 1 r 2 3 r 1 6 r 4 v -1 7 r -2 2 r -1 1 r -2 2 r s n 817 553 m 1 -2 r -1 -1 r -1 1 r 2 v 1 2 r 1 1 r 4 2 r 5 h 4 -2 r 1 -1 r 2 -2 r -3 v -2 -2 r -3 -3 r -7 -3 r -2 -1 r -3 -2 r -1 -4 r -4 v s n 827 558 m 3 -2 r 1 -1 r 1 -2 r -3 v -1 -2 r -4 -3 r -5 -3 r s n 816 533 m 1 2 r 3 h 6 -3 r 4 h 2 1 r 2 2 r s n 820 535 m 6 -4 r 5 h 1 1 r 2 3 r 2 v s n 849 558 m -4 -2 r -2 -3 r 841 546 l -4 v 843 536 l 2 -4 r 4 -1 r 3 h 4 1 r 2 4 r 1 6 r 4 v -1 7 r -2 3 r -4 2 r c s n 849 558 m -2 -2 r -2 -1 r -1 -2 r 843 546 l -4 v 844 536 l 1 -3 r 2 -1 r 2 -1 r s n 852 531 m 2 1 r 2 1 r 1 3 r 1 6 r 4 v -1 7 r -1 2 r -2 1 r -2 2 r s n 875 558 m -4 -2 r -3 -3 r 867 546 l -4 v 868 536 l 3 -4 r 4 -1 r 2 h 4 1 r 3 4 r 1 6 r 4 v -1 7 r -3 3 r -4 2 r c s n 875 558 m -3 -2 r -1 -1 r -1 -2 r 868 546 l -4 v 870 536 l 1 -3 r 1 -1 r 3 -1 r s n 877 531 m 3 1 r 1 1 r 2 3 r 1 6 r 4 v -1 7 r -2 2 r -1 1 r -3 2 r s -24 1108 555 d -27 1110 558 d n 1110 558 m 1096 539 l 20 h s 9 1105 531 e n 1130 558 m -4 -2 r -2 -3 r 1123 546 l -4 v 1124 536 l 2 -4 r 4 -1 r 3 h 4 1 r 2 4 r 2 6 r 4 v -2 7 r -2 3 r -4 2 r c s n 1130 558 m -2 -2 r -2 -1 r -1 -2 r 1124 546 l -4 v 1125 536 l 1 -3 r 2 -1 r 2 -1 r s n 1133 531 m 2 1 r 2 1 r 1 3 r 1 6 r 4 v -1 7 r -1 2 r -2 1 r -2 2 r s n 1156 558 m -4 -2 r -2 -3 r 1148 546 l -4 v 1150 536 l 2 -4 r 4 -1 r 3 h 3 1 r 3 4 r 1 6 r 4 v -1 7 r -3 3 r -3 2 r c s n 1156 558 m -3 -2 r -1 -1 r -1 -2 r 1150 546 l -4 v 1151 536 l 1 -3 r 1 -1 r 3 -1 r s n 1159 531 m 2 1 r 1 1 r 2 3 r 1 6 r 4 v -1 7 r -2 2 r -1 1 r -2 2 r s n 1393 554 m -1 -1 r 1 -2 r 2 2 r 1 v -2 2 r -2 2 r -4 h -4 -2 r -2 -2 r -2 -3 r -1 -5 r -7 v 1 -4 r 3 -3 r 4 -1 r 2 h 4 1 r 3 3 r 1 4 r 1 v -1 4 r -3 2 r -4 2 r -1 h -4 -2 r -2 -2 r -2 -4 r s n 1387 558 m -3 -2 r -2 -2 r -1 -3 r -2 -5 r -7 v 2 -4 r 2 -3 r 3 -1 r s n 1388 531 m 3 1 r 2 3 r 2 4 r 1 v -2 4 r -2 2 r -3 2 r s n 1411 558 m -4 -2 r -2 -3 r 1404 546 l -4 v 1405 536 l 2 -4 r 4 -1 r 3 h 4 1 r 2 4 r 2 6 r 4 v -2 7 r -2 3 r -4 2 r c s n 1411 558 m -2 -2 r -2 -1 r -1 -2 r 1405 546 l -4 v 1406 536 l 1 -3 r 2 -1 r 2 -1 r s n 1414 531 m 2 1 r 2 1 r 1 3 r 1 6 r 4 v -1 7 r -1 2 r -2 1 r -2 2 r s n 1437 558 m -4 -2 r -2 -3 r 1429 546 l -4 v 1431 536 l 2 -4 r 4 -1 r 3 h 3 1 r 3 4 r 1 6 r 4 v -1 7 r -3 3 r -3 2 r c s n 1437 558 m -3 -2 r -1 -1 r -1 -2 r 1431 546 l -4 v 1432 536 l 1 -3 r 1 -1 r 3 -1 r s n 1440 531 m 2 1 r 1 1 r 2 3 r 1 6 r 4 v -1 7 r -2 2 r -1 1 r -2 2 r s n 1665 558 m -3 -2 r -2 -2 r -4 v 2 -2 r 3 -2 r 6 h 3 2 r 2 2 r 4 v -2 2 r -3 2 r c s n 1665 558 m -2 -2 r -1 -2 r -4 v 1 -2 r 2 -2 r s n 1671 546 m 2 2 r 1 2 r 4 v -1 2 r -2 2 r s n 1665 546 m -3 -1 r -2 -1 r -1 -3 r -5 v 1 -3 r 2 -1 r 3 -1 r 6 h 3 1 r 2 1 r 1 3 r 5 v -1 3 r -2 1 r -3 1 r s n 1665 546 m -2 -1 r -1 -1 r -2 -3 r -5 v 2 -3 r 1 -1 r 2 -1 r s n 1671 531 m 2 1 r 1 1 r 2 3 r 5 v -2 3 r -1 1 r -2 1 r s n 1692 558 m -3 -2 r -3 -3 r 1685 546 l -4 v 1686 536 l 3 -4 r 3 -1 r 3 h 4 1 r 2 4 r 2 6 r 4 v -2 7 r -2 3 r -4 2 r c s n 1692 558 m -2 -2 r -1 -1 r -2 -2 r 1686 546 l -4 v 1687 536 l 2 -3 r 1 -1 r 2 -1 r s n 1695 531 m 3 1 r 1 1 r 1 3 r 1 6 r 4 v -1 7 r -1 2 r -1 1 r -3 2 r s n 1718 558 m -4 -2 r -2 -3 r 1710 546 l -4 v 1712 536 l 2 -4 r 4 -1 r 3 h 4 1 r 2 4 r 1 6 r 4 v -1 7 r -2 3 r -4 2 r c s n 1718 558 m -2 -2 r -2 -1 r -1 -2 r 1712 546 l -4 v 1713 536 l 1 -3 r 2 -1 r 2 -1 r s n 1721 531 m 2 1 r 2 1 r 1 3 r 1 6 r 4 v -1 7 r -1 2 r -2 1 r -2 2 r s n 1931 553 m 3 1 r 4 4 r -27 v s -25 1936 556 d 12 1931 531 e n 1961 558 m -4 -2 r -3 -3 r 1953 546 l -4 v 1954 536 l 3 -4 r 4 -1 r 2 h 4 1 r 3 4 r 1 6 r 4 v -1 7 r -3 3 r -4 2 r c s n 1961 558 m -3 -2 r -1 -1 r -1 -2 r 1954 546 l -4 v 1956 536 l 1 -3 r 1 -1 r 3 -1 r s n 1963 531 m 3 1 r 1 1 r 1 3 r 2 6 r 4 v -2 7 r -1 2 r -1 1 r -3 2 r s n 1986 558 m -3 -2 r -3 -3 r 1979 546 l -4 v 1980 536 l 3 -4 r 3 -1 r 3 h 4 1 r 2 4 r 2 6 r 4 v -2 7 r -2 3 r -4 2 r c s n 1986 558 m -2 -2 r -1 -1 r -2 -2 r 1980 546 l -4 v 1981 536 l 2 -3 r 1 -1 r 2 -1 r s n 1989 531 m 3 1 r 1 1 r 1 3 r 1 6 r 4 v -1 7 r -1 2 r -1 1 r -3 2 r s n 2012 558 m -4 -2 r -2 -3 r 2004 546 l -4 v 2006 536 l 2 -4 r 4 -1 r 3 h 4 1 r 2 4 r 1 6 r 4 v -1 7 r -2 3 r -4 2 r c s n 2012 558 m -2 -2 r -2 -1 r -1 -2 r 2006 546 l -4 v 2007 536 l 1 -3 r 2 -1 r 2 -1 r s n 2015 531 m 2 1 r 2 1 r 1 3 r 1 6 r 4 v -1 7 r -1 2 r -2 1 r -2 2 r s 1440 569 569 d 36 569 569 e 18 569 713 e 36 569 857 e 18 569 1001 e 36 569 1145 e 18 569 1289 e 36 569 1433 e 18 569 1577 e 36 569 1721 e 18 569 1865 e 36 569 2009 e n 528 585 m -4 -2 r -2 -3 r 521 573 l -4 v 522 563 l 2 -4 r 4 -1 r 3 h 4 1 r 2 4 r 2 6 r 4 v -2 7 r -2 3 r -4 2 r c s n 528 585 m -2 -2 r -2 -1 r -1 -2 r 522 573 l -4 v 523 563 l 1 -3 r 2 -1 r 2 -1 r s n 531 558 m 2 1 r 2 1 r 1 3 r 1 6 r 4 v -1 7 r -1 2 r -2 1 r -2 2 r s n 393 867 m 2 -1 r -2 -1 r -1 1 r 1 v 1 3 r 2 1 r 3 2 r 6 h 3 -2 r 2 -1 r 1 -3 r -2 v -1 -3 r -4 -2 r 398 857 l -2 -1 r -3 -3 r -1 -4 r -3 v s n 404 873 m 2 -2 r 1 -1 r 2 -3 r -2 v -2 -3 r -3 -2 r 398 857 l s n 392 848 m 1 1 r 3 h 6 -2 r 4 h 3 1 r 1 1 r s n 396 849 m 6 -3 r 5 h 2 1 r 1 2 r 3 v s n 425 873 m -3 -2 r -3 -4 r -1 -6 r -4 v 1 -6 r 3 -4 r 3 -1 r 3 h 4 1 r 2 4 r 2 6 r 4 v -2 6 r -2 4 r -4 2 r c s n 425 873 m -2 -2 r -1 -1 r -2 -3 r -1 -6 r -4 v 1 -6 r 2 -3 r 1 -1 r 2 -1 r s n 428 846 m 3 1 r 1 1 r 1 3 r 1 6 r 4 v -1 6 r -1 3 r -1 1 r -3 2 r s n 451 873 m -4 -2 r -2 -4 r -2 -6 r -4 v 2 -6 r 2 -4 r 4 -1 r 3 h 4 1 r 2 4 r 1 6 r 4 v -1 6 r -2 4 r -4 2 r c s n 451 873 m -2 -2 r -2 -1 r -1 -3 r -1 -6 r -4 v 1 -6 r 1 -3 r 2 -1 r 2 -1 r s n 454 846 m 2 1 r 2 1 r 1 3 r 1 6 r 4 v -1 6 r -1 3 r -2 1 r -2 2 r s n 477 873 m -4 -2 r -3 -4 r -1 -6 r -4 v 1 -6 r 3 -4 r 4 -1 r 2 h 4 1 r 3 4 r 1 6 r 4 v -1 6 r -3 4 r -4 2 r c s n 477 873 m -3 -2 r -1 -1 r -1 -3 r -2 -6 r -4 v 2 -6 r 1 -3 r 1 -1 r 3 -1 r s n 479 846 m 3 1 r 1 1 r 2 3 r 1 6 r 4 v -1 6 r -2 3 r -1 1 r -3 2 r s n 503 873 m -4 -2 r -3 -4 r -1 -6 r -4 v 1 -6 r 3 -4 r 4 -1 r 2 h 4 1 r 3 4 r 1 6 r 4 v -1 6 r -3 4 r -4 2 r c s n 503 873 m -3 -2 r -1 -1 r -2 -3 r -1 -6 r -4 v 1 -6 r 2 -3 r 1 -1 r 3 -1 r s n 505 846 m 3 1 r 1 1 r 1 3 r 2 6 r 4 v -2 6 r -1 3 r -1 1 r -3 2 r s n 528 873 m -4 -2 r -2 -4 r -1 -6 r -4 v 1 -6 r 2 -4 r 4 -1 r 3 h 4 1 r 2 4 r 2 6 r 4 v -2 6 r -2 4 r -4 2 r c s n 528 873 m -2 -2 r -2 -1 r -1 -3 r -1 -6 r -4 v 1 -6 r 1 -3 r 2 -1 r 2 -1 r s n 531 846 m 2 1 r 2 1 r 1 3 r 1 6 r 4 v -1 6 r -1 3 r -2 1 r -2 2 r s -25 404 1158 d -27 405 1160 d n 405 1160 m 391 1141 l 20 h s 9 400 1133 e n 425 1160 m -3 -1 r -3 -4 r -1 -6 r -4 v 1 -6 r 3 -4 r 3 -2 r 3 h 4 2 r 2 4 r 2 6 r 4 v -2 6 r -2 4 r -4 1 r c s n 425 1160 m -2 -1 r -1 -1 r -2 -3 r -1 -6 r -4 v 1 -6 r 2 -3 r 1 -1 r 2 -2 r s n 428 1133 m 3 2 r 1 1 r 1 3 r 1 6 r 4 v -1 6 r -1 3 r -1 1 r -3 1 r s n 451 1160 m -4 -1 r -2 -4 r -2 -6 r -4 v 2 -6 r 2 -4 r 4 -2 r 3 h 4 2 r 2 4 r 1 6 r 4 v -1 6 r -2 4 r -4 1 r c s n 451 1160 m -2 -1 r -2 -1 r -1 -3 r -1 -6 r -4 v 1 -6 r 1 -3 r 2 -1 r 2 -2 r s n 454 1133 m 2 2 r 2 1 r 1 3 r 1 6 r 4 v -1 6 r -1 3 r -2 1 r -2 1 r s n 477 1160 m -4 -1 r -3 -4 r -1 -6 r -4 v 1 -6 r 3 -4 r 4 -2 r 2 h 4 2 r 3 4 r 1 6 r 4 v -1 6 r -3 4 r -4 1 r c s n 477 1160 m -3 -1 r -1 -1 r -1 -3 r -2 -6 r -4 v 2 -6 r 1 -3 r 1 -1 r 3 -2 r s n 479 1133 m 3 2 r 1 1 r 2 3 r 1 6 r 4 v -1 6 r -2 3 r -1 1 r -3 1 r s n 503 1160 m -4 -1 r -3 -4 r -1 -6 r -4 v 1 -6 r 3 -4 r 4 -2 r 2 h 4 2 r 3 4 r 1 6 r 4 v -1 6 r -3 4 r -4 1 r c s n 503 1160 m -3 -1 r -1 -1 r -2 -3 r -1 -6 r -4 v 1 -6 r 2 -3 r 1 -1 r 3 -2 r s n 505 1133 m 3 2 r 1 1 r 1 3 r 2 6 r 4 v -2 6 r -1 3 r -1 1 r -3 1 r s n 528 1160 m -4 -1 r -2 -4 r -1 -6 r -4 v 1 -6 r 2 -4 r 4 -2 r 3 h 4 2 r 2 4 r 2 6 r 4 v -2 6 r -2 4 r -4 1 r c s n 528 1160 m -2 -1 r -2 -1 r -1 -3 r -1 -6 r -4 v 1 -6 r 1 -3 r 2 -1 r 2 -2 r s n 531 1133 m 2 2 r 2 1 r 1 3 r 1 6 r 4 v -1 6 r -1 3 r -2 1 r -2 1 r s n 407 1444 m -1 -1 r 1 -1 r 2 1 r 1 v -2 3 r -2 1 r -4 h 397 1447 l -2 -3 r -2 -2 r -1 -5 r -8 v 1 -4 r 3 -2 r 4 -2 r 2 h 4 2 r 3 2 r 1 4 r 1 v -1 4 r -3 3 r -4 1 r -1 h 397 1437 l -2 -3 r -2 -4 r s n 401 1448 m -3 -1 r -2 -3 r -1 -2 r -2 -5 r -8 v 2 -4 r 2 -2 r 3 -2 r s n 402 1421 m 3 2 r 2 2 r 2 4 r 1 v -2 4 r -2 3 r -3 1 r s n 425 1448 m -3 -1 r -3 -4 r -1 -6 r -4 v 1 -7 r 3 -3 r 3 -2 r 3 h 4 2 r 2 3 r 2 7 r 4 v -2 6 r -2 4 r -4 1 r c s n 425 1448 m -2 -1 r -1 -1 r -2 -3 r -1 -6 r -4 v 1 -7 r 2 -2 r 1 -1 r 2 -2 r s n 428 1421 m 3 2 r 1 1 r 1 2 r 1 7 r 4 v -1 6 r -1 3 r -1 1 r -3 1 r s n 451 1448 m -4 -1 r -2 -4 r -2 -6 r -4 v 2 -7 r 2 -3 r 4 -2 r 3 h 4 2 r 2 3 r 1 7 r 4 v -1 6 r -2 4 r -4 1 r c s n 451 1448 m -2 -1 r -2 -1 r -1 -3 r -1 -6 r -4 v 1 -7 r 1 -2 r 2 -1 r 2 -2 r s n 454 1421 m 2 2 r 2 1 r 1 2 r 1 7 r 4 v -1 6 r -1 3 r -2 1 r -2 1 r s n 477 1448 m -4 -1 r -3 -4 r -1 -6 r -4 v 1 -7 r 3 -3 r 4 -2 r 2 h 4 2 r 3 3 r 1 7 r 4 v -1 6 r -3 4 r -4 1 r c s n 477 1448 m -3 -1 r -1 -1 r -1 -3 r -2 -6 r -4 v 2 -7 r 1 -2 r 1 -1 r 3 -2 r s n 479 1421 m 3 2 r 1 1 r 2 2 r 1 7 r 4 v -1 6 r -2 3 r -1 1 r -3 1 r s n 503 1448 m -4 -1 r -3 -4 r -1 -6 r -4 v 1 -7 r 3 -3 r 4 -2 r 2 h 4 2 r 3 3 r 1 7 r 4 v -1 6 r -3 4 r -4 1 r c s n 503 1448 m -3 -1 r -1 -1 r -2 -3 r -1 -6 r -4 v 1 -7 r 2 -2 r 1 -1 r 3 -2 r s n 505 1421 m 3 2 r 1 1 r 1 2 r 2 7 r 4 v -2 6 r -1 3 r -1 1 r -3 1 r s n 528 1448 m -4 -1 r -2 -4 r -1 -6 r -4 v 1 -7 r 2 -3 r 4 -2 r 3 h 4 2 r 2 3 r 2 7 r 4 v -2 6 r -2 4 r -4 1 r c s n 528 1448 m -2 -1 r -2 -1 r -1 -3 r -1 -6 r -4 v 1 -7 r 1 -2 r 2 -1 r 2 -2 r s n 531 1421 m 2 2 r 2 1 r 1 2 r 1 7 r 4 v -1 6 r -1 3 r -2 1 r -2 1 r s n 398 1736 m -3 -1 r -2 -3 r -4 v 2 -2 r 3 -1 r 6 h 3 1 r 2 2 r 4 v -2 3 r -3 1 r c s n 398 1736 m -2 -1 r -1 -3 r -4 v 1 -2 r 2 -1 r s n 404 1725 m 2 1 r 1 2 r 4 v -1 3 r -2 1 r s n 398 1725 m -3 -2 r -2 -1 r -1 -3 r -5 v 1 -2 r 2 -2 r 3 -1 r 6 h 3 1 r 2 2 r 1 2 r 5 v -1 3 r -2 1 r -3 2 r s n 398 1725 m -2 -2 r -1 -1 r -2 -3 r -5 v 2 -2 r 1 -2 r 2 -1 r s n 404 1709 m 2 1 r 1 2 r 2 2 r 5 v -2 3 r -1 1 r -2 2 r s n 425 1736 m -3 -1 r -3 -4 r -1 -6 r -4 v 1 -7 r 3 -4 r 3 -1 r 3 h 4 1 r 2 4 r 2 7 r 4 v -2 6 r -2 4 r -4 1 r c s n 425 1736 m -2 -1 r -1 -1 r -2 -3 r -1 -6 r -4 v 1 -7 r 2 -2 r 1 -2 r 2 -1 r s n 428 1709 m 3 1 r 1 2 r 1 2 r 1 7 r 4 v -1 6 r -1 3 r -1 1 r -3 1 r s n 451 1736 m -4 -1 r -2 -4 r -2 -6 r -4 v 2 -7 r 2 -4 r 4 -1 r 3 h 4 1 r 2 4 r 1 7 r 4 v -1 6 r -2 4 r -4 1 r c s n 451 1736 m -2 -1 r -2 -1 r -1 -3 r -1 -6 r -4 v 1 -7 r 1 -2 r 2 -2 r 2 -1 r s n 454 1709 m 2 1 r 2 2 r 1 2 r 1 7 r 4 v -1 6 r -1 3 r -2 1 r -2 1 r s n 477 1736 m -4 -1 r -3 -4 r -1 -6 r -4 v 1 -7 r 3 -4 r 4 -1 r 2 h 4 1 r 3 4 r 1 7 r 4 v -1 6 r -3 4 r -4 1 r c s n 477 1736 m -3 -1 r -1 -1 r -1 -3 r -2 -6 r -4 v 2 -7 r 1 -2 r 1 -2 r 3 -1 r s n 479 1709 m 3 1 r 1 2 r 2 2 r 1 7 r 4 v -1 6 r -2 3 r -1 1 r -3 1 r s n 503 1736 m -4 -1 r -3 -4 r -1 -6 r -4 v 1 -7 r 3 -4 r 4 -1 r 2 h 4 1 r 3 4 r 1 7 r 4 v -1 6 r -3 4 r -4 1 r c s n 503 1736 m -3 -1 r -1 -1 r -2 -3 r -1 -6 r -4 v 1 -7 r 2 -2 r 1 -2 r 3 -1 r s n 505 1709 m 3 1 r 1 2 r 1 2 r 2 7 r 4 v -2 6 r -1 3 r -1 1 r -3 1 r s n 528 1736 m -4 -1 r -2 -4 r -1 -6 r -4 v 1 -7 r 2 -4 r 4 -1 r 3 h 4 1 r 2 4 r 2 7 r 4 v -2 6 r -2 4 r -4 1 r c s n 528 1736 m -2 -1 r -2 -1 r -1 -3 r -1 -6 r -4 v 1 -7 r 1 -2 r 2 -2 r 2 -1 r s n 531 1709 m 2 1 r 2 2 r 1 2 r 1 7 r 4 v -1 6 r -1 3 r -2 1 r -2 1 r s n 370 2019 m 3 1 r 4 4 r -27 v s -26 375 2023 d 12 370 1997 e n 400 2024 m 396 2023 l -3 -4 r -1 -7 r -3 v 1 -7 r 3 -4 r 4 -1 r 2 h 4 1 r 3 4 r 1 7 r 3 v -1 7 r -3 4 r -4 1 r c s n 400 2024 m -3 -1 r -1 -2 r -1 -2 r -2 -7 r -3 v 2 -7 r 1 -2 r 1 -2 r 3 -1 r s n 402 1997 m 3 1 r 1 2 r 1 2 r 2 7 r 3 v -2 7 r -1 2 r -1 2 r -3 1 r s n 425 2024 m -3 -1 r -3 -4 r -1 -7 r -3 v 1 -7 r 3 -4 r 3 -1 r 3 h 4 1 r 2 4 r 2 7 r 3 v -2 7 r -2 4 r -4 1 r c s n 425 2024 m -2 -1 r -1 -2 r -2 -2 r -1 -7 r -3 v 1 -7 r 2 -2 r 1 -2 r 2 -1 r s n 428 1997 m 3 1 r 1 2 r 1 2 r 1 7 r 3 v -1 7 r -1 2 r -1 2 r -3 1 r s n 451 2024 m -4 -1 r -2 -4 r -2 -7 r -3 v 2 -7 r 2 -4 r 4 -1 r 3 h 4 1 r 2 4 r 1 7 r 3 v -1 7 r -2 4 r -4 1 r c s n 451 2024 m -2 -1 r -2 -2 r -1 -2 r -1 -7 r -3 v 1 -7 r 1 -2 r 2 -2 r 2 -1 r s n 454 1997 m 2 1 r 2 2 r 1 2 r 1 7 r 3 v -1 7 r -1 2 r -2 2 r -2 1 r s n 477 2024 m -4 -1 r -3 -4 r -1 -7 r -3 v 1 -7 r 3 -4 r 4 -1 r 2 h 4 1 r 3 4 r 1 7 r 3 v -1 7 r -3 4 r -4 1 r c s n 477 2024 m -3 -1 r -1 -2 r -1 -2 r -2 -7 r -3 v 2 -7 r 1 -2 r 1 -2 r 3 -1 r s n 479 1997 m 3 1 r 1 2 r 2 2 r 1 7 r 3 v -1 7 r -2 2 r -1 2 r -3 1 r s n 503 2024 m -4 -1 r -3 -4 r -1 -7 r -3 v 1 -7 r 3 -4 r 4 -1 r 2 h 4 1 r 3 4 r 1 7 r 3 v -1 7 r -3 4 r -4 1 r c s n 503 2024 m -3 -1 r -1 -2 r -2 -2 r -1 -7 r -3 v 1 -7 r 2 -2 r 1 -2 r 3 -1 r s n 505 1997 m 3 1 r 1 2 r 1 2 r 2 7 r 3 v -2 7 r -1 2 r -1 2 r -3 1 r s n 528 2024 m -4 -1 r -2 -4 r -1 -7 r -3 v 1 -7 r 2 -4 r 4 -1 r 3 h 4 1 r 2 4 r 2 7 r 3 v -2 7 r -2 4 r -4 1 r c s n 528 2024 m -2 -1 r -2 -2 r -1 -2 r -1 -7 r -3 v 1 -7 r 1 -2 r 2 -2 r 2 -1 r s n 531 1997 m 2 1 r 2 2 r 1 2 r 1 7 r 3 v -1 7 r -1 2 r -2 2 r -2 1 r s 1440 569 2009 e -36 569 2009 d -18 710 2009 d -36 850 2009 d -18 991 2009 d -36 1132 2009 d -18 1272 2009 d -36 1413 2009 d -18 1553 2009 d -36 1694 2009 d -18 1834 2009 d -36 1975 2009 d 1440 2009 569 d -36 2009 569 e -18 2009 713 e -36 2009 857 e -18 2009 1001 e -36 2009 1145 e -18 2009 1289 e -36 2009 1433 e -18 2009 1577 e -36 2009 1721 e -18 2009 1865 e -36 2009 2009 e -27 1139 510 d -27 1141 510 d 6 1135 510 e 9 1135 483 e n 1152 494 m 16 h 2 v -2 3 r -1 1 r -3 1 r -3 h -4 -1 r -3 -3 r -1 -3 r -3 v 1 -4 r 3 -2 r 4 -2 r 2 h 4 2 r 3 2 r s n 1166 494 m 3 v -1 3 r s n 1159 501 m -3 -1 r -3 -3 r -1 -3 r -3 v 1 -4 r 3 -2 r 3 -2 r s -18 1178 501 d -18 1179 501 d n 1179 497 m 3 3 r 4 1 r 2 h 4 -1 r 1 -3 r -14 v s n 1188 501 m 3 -1 r 1 -3 r -14 v s 5 1174 501 e 9 1174 483 e 9 1188 483 e n 1210 501 m -3 -1 r -1 -1 r -1 -3 r -2 v 1 -3 r 1 -1 r 3 -2 r 2 h 3 2 r 1 1 r 2 3 r 2 v -2 3 r -1 1 r -3 1 r c s n 1207 500 m -1 -3 r -5 v 1 -2 r s n 1215 490 m 1 2 r 5 v -1 3 r s n 1216 499 m 2 1 r 2 1 r -1 v -2 h s n 1206 491 m -1 -1 r -2 -3 r -1 v 2 -3 r 4 -1 r 6 h 4 -1 r 1 -2 r s n 1203 486 m 2 -1 r 4 -2 r 6 h 4 -1 r 1 -3 r -1 v -1 -2 r -4 -2 r -8 h -4 2 r -1 2 r 1 v 1 3 r 4 1 r s n 1230 510 m -22 v 2 -3 r 2 -2 r 3 h 2 2 r 2 2 r s n 1232 510 m -22 v 1 -3 r 1 -2 r s 10 1227 501 e -27 1250 510 d -27 1251 510 d n 1251 497 m 3 3 r 3 1 r 3 h 4 -1 r 1 -3 r -14 v s n 1260 501 m 3 -1 r 1 -3 r -14 v s 5 1246 510 e 9 1246 483 e 9 1260 483 e n 1306 515 m -2 -2 r -3 -4 r -2 -5 r 1297 497 l -5 v 1299 486 l 1301 481 l 3 -4 r 2 -3 r s n 1304 513 m -3 -5 r -1 -4 r 1299 497 l -5 v 1300 486 l 1 -4 r 1304 477 l s -27 1317 510 d -27 1318 510 d n 1318 497 m 2 3 r 3 1 r 3 h 3 -1 r 3 -3 r 1 -3 r -3 v -1 -4 r -3 -2 r -3 -2 r -3 h -3 2 r -2 2 r s n 1326 501 m 2 -1 r 3 -3 r 1 -3 r -3 v -1 -4 r -3 -2 r -2 -2 r s 5 1313 510 e 8 -18 1342 501 i 6 -15 1344 501 i n 1358 501 m 1350 483 l 1347 478 l -2 -2 r -3 -2 r -1 h -1 2 r 1 1 r 1 -1 r s 7 1340 501 e 7 1353 501 e n 1368 510 m -22 v 1 -3 r 3 -2 r 2 h 3 2 r 1 2 r s n 1369 510 m -22 v 2 -3 r 1 -2 r s 10 1364 501 e n 1386 494 m 15 h 2 v -1 3 r -1 1 r -3 1 r -4 h -3 -1 r -3 -3 r -1 -3 r -3 v 1 -4 r 3 -2 r 3 -2 r 3 h 4 2 r 2 2 r s n 1400 494 m 3 v -1 3 r s n 1392 501 m -2 -1 r -3 -3 r -1 -3 r -3 v 1 -4 r 3 -2 r 2 -2 r s n 1422 499 m 1 2 r -5 v -1 3 r -1 1 r -3 1 r -5 h -3 -1 r -1 -1 r -3 v 1 -1 r 3 -1 r 6 -3 r 3 -1 r 1 -2 r s n 1409 497 m 1 -1 r 3 -1 r 6 -3 r 3 -1 r 1 -1 r -4 v -1 -1 r -3 -2 r -5 h -2 2 r -2 1 r -1 2 r -5 v 1 3 r s n 1431 515 m 3 -2 r 2 -4 r 3 -5 r 1440 497 l -5 v 1439 486 l 1436 481 l -2 -4 r -3 -3 r s n 1434 513 m 2 -5 r 1 -4 r 1439 497 l -5 v 1437 486 l -1 -4 r 1434 477 l s n 870 2060 m 2 -5 r 10 v -2 -5 r -3 3 r -6 2 r -3 h -5 -2 r -4 -3 r -1 -3 r -2 -6 r -8 v 2 -5 r 1 -4 r 4 -3 r 5 -2 r 3 h 6 2 r 3 3 r 2 4 r s n 858 2065 m -3 -2 r -4 -3 r -2 -3 r -1 -6 r -8 v 1 -5 r 2 -4 r 4 -3 r 3 -2 r s n 892 2053 m -5 -2 r -3 -3 r -2 -5 r -4 v 2 -5 r 3 -3 r 5 -2 r 4 h 5 2 r 3 3 r 2 5 r 4 v -2 5 r -3 3 r -5 2 r c s n 892 2053 m -3 -2 r -4 -3 r -1 -5 r -4 v 1 -5 r 4 -3 r 3 -2 r s n 896 2029 m 3 2 r 4 3 r 1 5 r 4 v -1 5 r -4 3 r -3 2 r s -24 920 2053 d -24 921 2053 d n 921 2048 m 4 3 r 5 2 r 3 h 6 -2 r 1 -3 r -19 v s n 933 2053 m 4 -2 r 2 -3 r -19 v s n 940 2048 m 4 3 r 5 2 r 3 h 5 -2 r 2 -3 r -19 v s n 952 2053 m 4 -2 r 1 -3 r -19 v s 6 915 2053 e 12 915 2029 e 12 933 2029 e 12 952 2029 e -24 976 2053 d -24 978 2053 d n 978 2048 m 3 3 r 6 2 r 3 h 5 -2 r 2 -3 r -19 v s n 990 2053 m 3 -2 r 2 -3 r -19 v s n 997 2048 m 3 3 r 5 2 r 4 h 5 -2 r 2 -3 r -19 v s n 1009 2053 m 3 -2 r 2 -3 r -19 v s 7 971 2053 e 12 971 2029 e 12 990 2029 e 12 1009 2029 e n 1033 2053 m -19 v 2 -3 r 5 -2 r 3 h 5 2 r 4 3 r s n 1035 2053 m -19 v 1 -3 r 4 -2 r s -24 1052 2053 d -24 1053 2053 d 7 1028 2053 e 6 1047 2053 e 7 1052 2029 e -24 1071 2053 d -24 1072 2053 d n 1072 2048 m 4 3 r 5 2 r 3 h 5 -2 r 2 -3 r -19 v s n 1084 2053 m 4 -2 r 1 -3 r -19 v s 7 1065 2053 e 12 1065 2029 e 12 1084 2029 e n 1108 2065 m -2 -2 r 2 -1 r 2 1 r c s -24 1108 2053 d -24 1110 2053 d 7 1103 2053 e 12 1103 2029 e n 1144 2048 m -2 -2 r 2 -1 r 2 1 r 2 v -4 3 r -3 2 r -5 h -5 -2 r -4 -3 r -1 -5 r -4 v 1 -5 r 4 -3 r 5 -2 r 3 h 5 2 r 4 3 r s n 1134 2053 m -4 -2 r -3 -3 r -2 -5 r -4 v 2 -5 r 3 -3 r 4 -2 r s n 1160 2050 m -2 v -2 h 2 v 2 1 r 3 2 r 7 h 3 -2 r 2 -1 r 2 -4 r -12 v 1 -3 r 2 -2 r s n 1175 2050 m -16 v 2 -3 r 3 -2 r 2 h s n 1175 2046 m -2 -1 r -10 -2 r -5 -2 r -2 -3 r -4 v 2 -3 r 5 -2 r 5 h 4 2 r 3 3 r s n 1163 2043 m -3 -2 r -2 -3 r -4 v 2 -3 r 3 -2 r s n 1194 2065 m -29 v 2 -5 r 3 -2 r 3 h 4 2 r 2 3 r s n 1196 2065 m -29 v 1 -5 r 2 -2 r s 13 1189 2053 e n 1220 2065 m -2 -2 r 2 -1 r 1 1 r c s -24 1220 2053 d -24 1221 2053 d 7 1214 2053 e 12 1214 2029 e n 1245 2053 m -5 -2 r -3 -3 r -2 -5 r -4 v 2 -5 r 3 -3 r 5 -2 r 4 h 5 2 r 3 3 r 2 5 r 4 v -2 5 r -3 3 r -5 2 r c s n 1245 2053 m -3 -2 r -4 -3 r -1 -5 r -4 v 1 -5 r 4 -3 r 3 -2 r s n 1249 2029 m 3 2 r 4 3 r 1 5 r 4 v -1 5 r -4 3 r -3 2 r s -24 1273 2053 d -24 1274 2053 d n 1274 2048 m 4 3 r 5 2 r 3 h 6 -2 r 1 -3 r -19 v s n 1286 2053 m 4 -2 r 2 -3 r -19 v s 6 1268 2053 e 12 1268 2029 e 12 1286 2029 e n 1324 2050 m 2 3 r -7 v -2 4 r -2 1 r -3 2 r -7 h -3 -2 r -2 -1 r -4 v 2 -1 r 3 -2 r 9 -4 r 3 -1 r 2 -2 r s n 1307 2048 m 2 -2 r 3 -1 r 9 -4 r 3 -2 r 2 -1 r -5 v -2 -2 r -3 -2 r -7 h -4 2 r -1 2 r -2 3 r -7 v 2 4 r s -36 1367 2065 d -36 1369 2065 d n 1362 2065 m 20 h 5 -2 r 2 -1 r 2 -4 r -5 v -2 -3 r -2 -2 r -5 -2 r -13 h s n 1382 2065 m 4 -2 r 1 -1 r 2 -4 r -5 v -2 -3 r -1 -2 r -4 -2 r s 12 1362 2029 e n 1403 2043 m 20 h 3 v -1 4 r -2 1 r -3 2 r -6 h -5 -2 r -3 -3 r -2 -5 r -4 v 2 -5 r 3 -3 r 5 -2 r 4 h 5 2 r 3 3 r s n 1422 2043 m 5 v -2 3 r s n 1411 2053 m -3 -2 r -3 -3 r -2 -5 r -4 v 2 -5 r 3 -3 r 3 -2 r s -24 1437 2053 d -24 1439 2053 d n 1439 2043 m 2 5 r 3 3 r 3 2 r 6 h 1 -2 r -1 v -1 -2 r -2 2 r 2 1 r s 7 1432 2053 e 12 1432 2029 e n 1475 2063 m -2 -1 r 2 -2 r 2 2 r 1 v -2 2 r -4 h -3 -2 r -2 -3 r -31 v s n 1471 2065 m -1 -2 r -2 -3 r -31 v s 14 1461 2053 e 12 1461 2029 e n 1495 2053 m -5 -2 r -3 -3 r -2 -5 r -4 v 2 -5 r 3 -3 r 5 -2 r 4 h 5 2 r 3 3 r 2 5 r 4 v -2 5 r -3 3 r -5 2 r c s n 1495 2053 m -3 -2 r -3 -3 r -2 -5 r -4 v 2 -5 r 3 -3 r 3 -2 r s n 1499 2029 m 3 2 r 4 3 r 1 5 r 4 v -1 5 r -4 3 r -3 2 r s -24 1523 2053 d -24 1525 2053 d n 1525 2043 m 1 5 r 4 3 r 3 2 r 5 h 2 -2 r -1 v -2 -2 r -1 2 r 1 1 r s 7 1518 2053 e 12 1518 2029 e -24 1552 2053 d -24 1554 2053 d n 1554 2048 m 3 3 r 5 2 r 4 h 5 -2 r 2 -3 r -19 v s n 1566 2053 m 3 -2 r 2 -3 r -19 v s n 1573 2048 m 3 3 r 5 2 r 3 h 6 -2 r 1 -3 r -19 v s n 1584 2053 m 4 -2 r 2 -3 r -19 v s 7 1547 2053 e 12 1547 2029 e 12 1566 2029 e 12 1584 2029 e n 1608 2050 m -2 v -1 h 2 v 1 1 r 4 2 r 7 h 3 -2 r 2 -1 r 2 -4 r -12 v 1 -3 r 2 -2 r s n 1624 2050 m -16 v 2 -3 r 3 -2 r 2 h s n 1624 2046 m -2 -1 r -10 -2 r -5 -2 r -2 -3 r -4 v 2 -3 r 5 -2 r 5 h 3 2 r 4 3 r s n 1612 2043 m -4 -2 r -1 -3 r -4 v 1 -3 r 4 -2 r s -24 1643 2053 d -24 1644 2053 d n 1644 2048 m 4 3 r 5 2 r 3 h 6 -2 r 1 -3 r -19 v s n 1656 2053 m 4 -2 r 2 -3 r -19 v s 6 1638 2053 e 12 1638 2029 e 12 1656 2029 e n 1698 2048 m -2 -2 r 2 -1 r 1 1 r 2 v -3 3 r -4 2 r -5 h -5 -2 r -3 -3 r -2 -5 r -4 v 2 -5 r 3 -3 r 5 -2 r 4 h 5 2 r 3 3 r s n 1687 2053 m -3 -2 r -4 -3 r -1 -5 r -4 v 1 -5 r 4 -3 r 3 -2 r s n 1711 2043 m 21 h 3 v -2 4 r -2 1 r -3 2 r -5 h -5 -2 r -4 -3 r -1 -5 r -4 v 1 -5 r 4 -3 r 5 -2 r 3 h 5 2 r 4 3 r s n 1730 2043 m 5 v -2 3 r s n 1720 2053 m -4 -2 r -3 -3 r -2 -5 r -4 v 2 -5 r 3 -3 r 4 -2 r s 27 320 1110 e 27 320 1112 e n 320 1106 m 16 v 1 4 r 2 1 r 2 1 r 3 h 2 -1 r 2 -1 r 1 -4 r -10 v s n 320 1122 m 1 2 r 2 2 r 2 1 r 3 h 2 -1 r 2 -2 r 1 -2 r s 9 347 1106 d n 333 1118 m 1 3 r 1 1 r 9 4 r 2 1 r 1 v -2 2 r s n 334 1121 m 3 1 r 9 2 r 1 2 r 2 v -3 2 r -1 h s n 332 1139 m 1 h -2 v -1 h -2 2 r -1 2 r 5 v 1 3 r 2 1 r 2 1 r 9 h 3 2 r 1 1 r s n 332 1150 m 11 h 3 1 r 1 3 r 1 v s n 334 1150 m 1 -1 r 2 -8 r 1 -4 r 3 -1 r 2 h 3 1 r 1 4 r 4 v -1 3 r -3 2 r s n 337 1141 m 1 -2 r 3 -2 r 2 h 3 2 r 1 2 r s n 320 1164 m 22 h 4 2 r 1 2 r 3 v -1 2 r -3 2 r s n 320 1166 m 22 h 4 1 r 1 1 r s 11 329 1160 d n 337 1182 m 16 v -3 h -2 -2 r -2 -1 r -1 -2 r -4 v 1 -4 r 3 -3 r 4 -1 r 2 h 4 1 r 3 3 r 1 4 r 2 v -1 4 r -3 3 r s n 337 1196 m -4 h -3 -1 r s n 329 1189 m 1 -3 r 3 -2 r 4 -2 r 2 h 4 2 r 3 2 r 1 3 r s n 315 1236 m 2 -2 r 4 -3 r 5 -2 r 7 -2 r 5 h 6 2 r 6 2 r 3 3 r 3 2 r s n 317 1234 m 6 -3 r 3 -1 r 7 -1 r 5 h 6 1 r 4 1 r 5 3 r s 27 320 1247 e 27 320 1248 e n 333 1248 m -3 2 r -1 3 r 3 v 1 3 r 3 3 r 4 1 r 2 h 4 -1 r 3 -3 r 1 -3 r -3 v -1 -3 r -3 -2 r s n 329 1256 m 1 2 r 3 3 r 4 1 r 2 h 4 -1 r 3 -3 r 1 -2 r s 5 320 1243 d 18 8 329 1272 i 15 6 329 1274 i n 329 1288 m 18 -8 r 5 -3 r 3 -2 r 1 -3 r -1 v -1 -1 r -2 1 r 2 1 r s 7 329 1270 d 7 329 1283 d n 320 1298 m 22 h 4 1 r 1 3 r 2 v -1 3 r -3 1 r s n 320 1299 m 22 h 4 2 r 1 1 r s 10 329 1294 d n 337 1316 m 15 v -3 h -2 -1 r -2 -1 r -1 -3 r -4 v 1 -3 r 3 -3 r 4 -1 r 2 h 4 1 r 3 3 r 1 3 r 3 v -1 4 r -3 2 r s n 337 1330 m -4 h -3 -1 r s n 329 1322 m 1 -2 r 3 -3 r 4 -1 r 2 h 4 1 r 3 3 r 1 2 r s n 332 1352 m -3 1 r 5 h -2 -1 r -2 -1 r -1 -3 r -5 v 1 -3 r 2 -1 r 2 h 1 1 r 2 3 r 2 6 r 2 3 r 1 1 r s n 333 1339 m 1 1 r 1 3 r 3 6 r 1 3 r 2 1 r 3 h 2 -1 r 1 -3 r -5 v -1 -2 r -2 -2 r -2 -1 r 5 h -3 1 r s 356 1360 315 1383 b n 332 1402 m -3 1 r 5 h -2 -1 r -2 -1 r -1 -3 r -5 v 1 -3 r 2 -1 r 2 h 1 1 r 2 3 r 2 6 r 2 3 r 1 1 r s n 333 1389 m 1 1 r 1 3 r 3 6 r 1 3 r 2 1 r 3 h 2 -1 r 1 -3 r -5 v -1 -2 r -2 -2 r -2 -1 r 5 h -3 1 r s n 337 1412 m 16 v -3 h -2 -2 r -2 -1 r -1 -2 r -4 v 1 -4 r 3 -3 r 4 -1 r 2 h 4 1 r 3 3 r 1 4 r 2 v -1 4 r -3 3 r s n 337 1426 m -4 h -3 -1 r s n 329 1419 m 1 -3 r 3 -2 r 4 -2 r 2 h 4 2 r 3 2 r 1 3 r s n 333 1451 m 1 -1 r 1 1 r -1 1 r -1 h -3 -2 r -1 -3 r -4 v 1 -4 r 3 -2 r 4 -2 r 2 h 4 2 r 3 2 r 1 4 r 3 v -1 4 r -3 2 r s n 329 1443 m 1 -2 r 3 -3 r 4 -1 r 2 h 4 1 r 3 3 r 1 2 r s n 315 1460 m 2 2 r 4 3 r 5 3 r 7 1 r 5 h 6 -1 r 6 -3 r 3 -3 r 3 -2 r s n 317 1462 m 6 3 r 3 1 r 7 2 r 5 h 6 -2 r 4 -1 r 5 -3 r s 657 974 614 806 b 20 -7 647 983 i 21 -7 647 984 i n 657 981 m -1 2 r 3 v 2 v 2 2 r 3 1 r 3 h 2 h 3 -2 r 1 -3 r -3 v -1 -2 r -2 -2 r -2 -1 r s n 656 988 m 2 1 r 3 1 r 3 h 2 h 2 -2 r 1 -3 r 1 -2 r s 1 4 646 980 i 21 -8 654 1004 i 20 -7 655 1004 i 2 3 653 1001 i 2 7 674 993 i n 666 1016 m -4 v 2 -2 r 2 -2 r 2 -1 r 3 h 3 1 r 2 3 r 1 2 r 3 v -2 2 r -2 2 r -2 1 r -3 h -3 -1 r -2 -3 r c s n 666 1016 m 1 -3 r 1 -2 r 2 -2 r 2 -1 r 4 h 2 1 r 2 2 r s n 681 1013 m -1 2 r -1 3 r -2 2 r -2 h -4 h -2 -1 r -2 -2 r s n 678 1040 m 1 -2 r 1 1 r 1 v -1 1 r -3 -2 r -2 -1 r -1 -3 r -3 v 2 -3 r 2 -2 r 2 -1 r 3 h 3 1 r 2 3 r 1 2 r 3 v -2 3 r s n 673 1035 m 1 -2 r 1 -3 r 2 -2 r 2 -1 r 4 h 2 1 r 2 2 r s 20 -8 672 1052 i 21 -8 672 1053 i 6 -13 683 1060 i 9 3 687 1053 i 10 3 686 1052 i 1 4 671 1049 i 3 6 681 1057 i 3 6 691 1042 i 697 1058 695 1052 b n 679 1072 m 1 -1 r 1 h 2 v c s 14 -5 686 1070 i 14 -5 686 1071 i 1 4 685 1067 i 701 1069 699 1062 b 13 -5 690 1080 i 14 -5 690 1081 i n 693 1080 m -1 3 r 3 v 1 2 r 2 3 r 2 h 11 -4 r s n 693 1088 m 1 2 r 3 h 10 -4 r s 1 4 689 1077 i 3 7 702 1072 i 709 1090 706 1083 b n 699 1104 m -2 v 1 -1 r 1 -2 r 2 -1 r 2 1 r 2 h 1 2 r 1 2 r 2 v -1 1 r -1 2 r -2 1 r -3 h -1 -1 r -2 -2 r c s n 699 1102 m 1 -1 r 4 -2 r 3 h s n 709 1105 m -2 2 r -4 1 r -2 h s n 702 1109 m 1 v -1 2 r 1 h -2 v s n 705 1099 m 1 -2 r 2 -1 r -1 v 3 1 r 2 2 r 2 5 r 2 3 r 1 h s n 708 1095 m 2 1 r 2 3 r 2 4 r 2 3 r 2 h 1 h 1 -2 r -3 v -2 -6 r -2 -2 r -2 -1 r -1 1 r -2 1 r 4 v s 41 -155 708 1114 i 7 10 749 959 i 18 -12 755 991 i 18 -12 756 992 i n 764 986 m 3 v 2 v 1 2 r 3 2 r 3 h 3 -1 r 1 -1 r 2 -2 r 1 -3 r -1 -3 r -1 -2 r -2 -1 r -3 -1 r s n 765 993 m 2 1 r 3 h 3 h 2 -2 r 2 -2 r -3 v -2 v s 756 992 754 988 b 18 -12 767 1009 i 18 -12 768 1010 i 2 4 766 1006 i 787 1000 783 994 b n 782 1018 m -1 -4 r 1 -2 r 2 -3 r 1 -1 r 3 -1 r 3 h 3 2 r 1 2 r 1 3 r -1 3 r -2 3 r -1 1 r -4 1 r -2 -1 r -3 -2 r c s n 782 1018 m -3 v -3 v 2 -2 r 2 -1 r 3 -1 r 3 h 2 1 r s n 795 1011 m 2 v 3 v -2 3 r -2 1 r -3 1 r -3 -1 r -2 -1 r s n 799 1038 m 1 -1 r 1 h 1 v -1 1 r -3 -1 r -2 -1 r -2 -2 r -1 -4 r 1 -2 r 2 -3 r 2 -1 r 3 -1 r 3 1 r 2 2 r 1 1 r 1 3 r 3 v s n 793 1035 m -3 v 1 -3 r 2 -2 r 1 -1 r 3 -1 r 3 h 2 2 r s 18 -12 796 1051 i 17 -12 797 1052 i 3 -15 808 1057 i 10 1 811 1049 i 10 1 810 1048 i 3 3 794 1049 i 3 5 807 1054 i 4 6 812 1037 i 822 1051 818 1046 b n 808 1069 m -1 v 2 h -1 2 r c s 12 -8 814 1065 i 12 -8 815 1066 i 3 3 812 1063 i 828 1061 824 1055 b 12 -8 820 1075 i 12 -9 821 1076 i n 823 1074 m 3 v 1 3 r 1 1 r 2 2 r 3 h 9 -6 r s n 825 1081 m 2 2 r 2 -1 r 10 -6 r s 2 4 819 1072 i 4 6 831 1064 i 841 1079 837 1073 b n 835 1096 m -1 -2 r 1 -2 r 1 -2 r 1 -1 r 3 h 1 h 2 1 r 1 2 r 1 2 r -1 2 r -1 2 r -1 1 r -3 h -1 h -2 -1 r c s n 834 1094 m 2 -2 r 3 -3 r 2 h s n 845 1094 m -1 2 r -4 3 r -2 h s n 839 1099 m 2 v 2 v 1 -1 r -1 -1 r s n 840 1089 m -2 v 1 -2 r 1 h 2 h 3 2 r 3 4 r 2 2 r 2 h s n 842 1085 m 1 h 3 2 r 3 4 r 2 2 r 3 h 1 -1 r 1 -2 r -1 -3 r -4 -5 r -2 -2 r -2 1 r -1 h -1 2 r 3 v s 32 42 852 1113 i 45 52 884 1155 i 36 36 929 1207 i 16 -15 968 1264 i 15 -15 969 1265 i n 976 1258 m 3 v 1 2 r 2 1 r 2 2 r 3 h 3 -2 r 2 -1 r 1 -3 r -3 v -1 -3 r -2 -1 r -2 -1 r -3 h s n 979 1264 m 2 1 r 3 h 3 -1 r 1 -2 r 1 -3 r -3 v -2 v s 969 1265 966 1262 b 16 -15 983 1280 i 16 -15 984 1280 i 3 3 981 1277 i 1002 1267 997 1262 b n 999 1286 m -1 -3 r -3 v 1 -3 r 2 -2 r 3 -1 r 3 h 3 2 r 1 1 r 2 3 r -1 3 r -1 3 r -1 1 r -3 2 r -3 h -3 -2 r c s n 999 1286 m -3 v -2 v 1 -3 r 2 -2 r 3 -1 r 2 h 3 1 r s n 1011 1277 m 1 2 r 3 v -2 3 r -1 1 r -3 2 r -3 h -2 -1 r s n 1020 1303 m -2 v 2 h 2 v -1 h -3 h -2 h -2 -3 r -2 -3 r -2 v 2 -3 r 1 -2 r 3 -1 r 3 h 3 1 r 1 2 r 2 3 r 3 v s n 1014 1300 m -1 -2 r -3 v 2 -3 r 1 -1 r 3 -2 r 3 1 r 2 h s 16 -15 1019 1316 i 16 -15 1020 1317 i 1 -14 1032 1319 i 10 -1 1033 1311 i 10 -2 1033 1311 i 3 3 1017 1314 i 5 5 1030 1317 i 5 5 1033 1299 i 1045 1312 1041 1307 b n 1035 1332 m -2 v 1 h 2 v c s 10 -10 1040 1327 i 11 -11 1040 1328 i 3 3 1037 1325 i 1053 1320 1048 1314 b 10 -10 1048 1335 i 11 -10 1048 1336 i n 1050 1333 m 3 v 2 3 r 1 2 r 3 1 r 2 h 8 -8 r s n 1053 1341 m 2 h 3 h 8 -8 r s 3 3 1045 1333 i 5 5 1056 1323 i 1069 1336 1064 1331 b n 1066 1353 m -1 -2 r -1 v 1 -3 r 1 -1 r 2 -1 r 2 h 2 1 r 1 1 r 1 3 r 1 v -1 2 r -1 2 r -2 h -2 h -2 h c s n 1065 1351 m 1 -2 r 3 -3 r 2 -1 r s n 1075 1350 m -1 2 r -3 3 r -2 h s n 1071 1355 m 2 v 2 v 1 -1 r -1 -1 r s n 1069 1345 m -1 v 1 -2 r 1 -1 r 2 -1 r 3 2 r 3 3 r 3 2 r 2 h s n 1071 1341 m 1 h 3 1 r 4 4 r 3 2 r 2 -1 r 1 -1 r -2 v -1 -3 r -4 -4 r -3 -2 r -3 1 r 1 v -1 2 r 1 3 r s 23 22 1086 1366 i 45 37 1109 1388 i 45 34 1154 1425 i 21 15 1199 1459 i 11 -19 1229 1494 i 11 -19 1230 1494 i n 1235 1485 m 1 3 r 1 2 r 2 1 r 3 1 r 3 -1 r 3 -2 r 1 -2 r -3 v -1 -3 r -2 -3 r -1 -1 r -3 h -3 1 r s n 1239 1491 m 3 h 2 -1 r 3 -2 r 1 -2 r -3 v -3 v -2 -2 r s 1230 1494 1227 1492 b 11 -19 1248 1504 i 11 -19 1249 1505 i 4 2 1245 1503 i 1262 1488 1256 1484 b n 1265 1506 m -2 -3 r -1 -3 r 1 -3 r 1 -2 r 2 -2 r 3 -1 r 3 1 r 2 1 r 2 3 r 1 2 r -1 4 r -1 1 r -2 2 r -3 1 r -3 h c s n 1265 1506 m -1 -2 r -1 -3 r 1 -3 r 1 -2 r 2 -2 r 3 -1 r 2 h s n 1274 1494 m 1 2 r 1 3 r -1 3 r -1 2 r -2 2 r -3 1 r -2 h s n 1290 1516 m -1 v 1 -1 r 2 v 1 v -3 h -2 h -3 -1 r -2 -3 r -1 -2 r 1 -4 r 1 -1 r 2 -3 r 3 h 3 h 2 1 r 2 3 r 1 2 r s n 1283 1516 m -1 -2 r -1 -3 r 1 -3 r 1 -2 r 2 -2 r 3 -1 r 2 h s 11 -19 1293 1530 i 10 -19 1294 1530 i -4 -14 1306 1529 i 9 -4 1305 1521 i 9 -5 1304 1521 i 4 2 1290 1528 i 5 3 1304 1527 i 6 4 1301 1509 i 1316 1518 1311 1515 b n 1312 1540 m -1 -1 r 2 -1 r 2 v c s 7 -13 1315 1534 i 7 -13 1316 1535 i 3 3 1313 1532 i 1326 1524 1320 1520 b 7 -13 1325 1540 i 7 -12 1326 1540 i n 1327 1537 m 1 3 r 2 3 r 2 1 r 3 h 2 -1 r 6 -10 r s n 1332 1544 m 2 h 2 -2 r 6 -9 r s 4 2 1322 1538 i 7 3 1329 1526 i 1346 1535 1339 1531 b n 1347 1552 m -1 -2 r -1 v -2 v 1 -2 r 2 -1 r 1 -1 r 2 h 2 1 r 2 2 r 2 v 2 v -1 2 r -2 1 r -2 h -2 h c s n 1346 1550 m -2 v 2 -4 r 2 -1 r s n 1356 1546 m -1 3 r -2 3 r -2 1 r s n 1353 1553 m 1 v 2 2 r -1 v -2 -1 r s n 1349 1544 m -1 -2 r 1 -2 r -1 v 2 -1 r 3 h 5 3 r 3 1 r 1 -1 r s n 1349 1539 m 1 h 4 h 4 3 r 3 h 2 -1 r 1 -1 r -2 v -2 -2 r -6 -4 r -3 h -2 1 r 1 v 2 v 2 3 r s 9 4 1370 1559 i 45 31 1379 1563 i 45 26 1424 1594 i 45 23 1469 1620 i 6 2 1514 1643 i 7 -20 1532 1663 i 7 -21 1533 1664 i n 1536 1654 m 1 3 r 2 1 r 2 1 r 3 h 3 -1 r 2 -3 r 1 -1 r -4 v -2 -2 r -2 -2 r -2 -1 r -2 h -3 1 r s n 1541 1659 m 2 h 3 -1 r 2 -3 r 1 -2 r -3 v -1 -3 r -2 -1 r s 1533 1664 1529 1662 b 7 -20 1552 1671 i 7 -20 1553 1671 i 4 1 1549 1670 i 1563 1652 1557 1650 b n 1569 1670 m -3 -2 r -1 -3 r -3 v 1 -2 r 2 -3 r 3 -1 r 3 h 2 1 r 2 2 r 1 3 r 3 v 2 v -2 2 r -3 2 r -3 -1 r c s n 1569 1670 m -2 -2 r -1 -3 r -3 v 1 -2 r 2 -2 r 3 -2 r 2 h s n 1576 1657 m 1 2 r 1 2 r 4 v 2 v -2 2 r -3 1 r -2 h s n 1595 1676 m -1 -1 r 1 -1 r 1 2 r 1 v -3 1 r -2 h -3 -1 r -3 -2 r -1 -3 r -3 v 1 -2 r 2 -2 r 3 -2 r 3 1 r 2 h 2 2 r 2 3 r s n 1588 1677 m -2 -2 r -1 -2 r -4 v 1 -1 r 2 -3 r 3 -1 r 2 h s 7 -20 1600 1689 i 7 -21 1601 1690 i -6 -14 1613 1687 i 9 -5 1610 1679 i 9 -6 1609 1679 i 4 2 1597 1688 i 6 3 1610 1685 i 6 3 1605 1668 i 1621 1674 1615 1672 b n 1620 1697 m -1 -1 r 2 -1 r 1 v c s 6 -13 1622 1690 i 6 -14 1623 1691 i 3 2 1620 1689 i 1631 1678 1625 1676 b 5 -13 1633 1694 i 5 -14 1634 1695 i n 1635 1692 m 1 2 r 3 2 r 2 1 r 3 h 2 -1 r 4 -11 r s n 1641 1697 m 2 h 2 -2 r 4 -10 r s 4 2 1630 1693 i 7 2 1635 1680 i 1653 1686 1646 1684 b n 1657 1703 m -2 -1 r -2 v -2 v -2 v 2 -1 r 1 -1 r 3 h 2 1 r 1 1 r 1 2 r 2 v -1 2 r -1 1 r -2 1 r -2 h c s n 1655 1702 m -3 v 2 -3 r 1 -2 r s n 1664 1696 m 3 v -1 3 r -2 2 r s n 1663 1703 m 2 v 2 1 r -1 v -2 h s n 1657 1695 m -1 -2 r -2 v 1 -1 r 1 -1 r 4 h 4 2 r 4 h 1 -1 r s n 1657 1690 m 1 h 3 h 5 1 r 3 1 r 2 -2 r -1 v -2 v -2 -2 r -6 -2 r -3 -1 r -2 2 r 1 v 2 v 2 2 r s 13 4 1681 1707 i 45 21 1694 1711 i 45 18 1739 1732 i 45 17 1784 1750 i 12 3 1829 1767 i 5 -21 1855 1787 i 5 -20 1856 1787 i n 1858 1777 m 2 3 r 1 2 r 2 h 4 h 2 -1 r 2 -3 r 1 -2 r -1 -3 r -1 -3 r -3 -2 r -2 h -2 h -2 2 r s n 1863 1782 m 3 h 2 -2 r 2 -2 r 1 -2 r -1 -4 r -1 -2 r -2 -2 r s 1856 1787 1852 1786 b 6 -21 1875 1793 i 6 -20 1876 1793 i 4 1 1872 1792 i 1885 1773 1878 1771 b n 1892 1790 m -3 -2 r -1 -2 r -3 v -2 v 2 -3 r 3 -1 r 3 -1 r 2 1 r 3 2 r 1 2 r 4 v 2 v -2 2 r -3 2 r -3 h c s n 1892 1790 m -2 -1 r -1 -3 r -3 v -2 v 2 -3 r 3 -1 r 2 -1 r s n 1898 1777 m 2 2 r 1 2 r 3 v 2 v -2 3 r -3 1 r -2 1 r s n 1919 1795 m -1 -2 r 1 h 1 1 r 1 v -3 1 r -2 1 r -3 -1 r -3 -2 r -1 -2 r -4 v -2 v 2 -2 r 3 -2 r 3 h 2 1 r 2 2 r 2 2 r s n 1912 1796 m -2 -2 r -1 -2 r -3 v -2 v 2 -3 r 3 -1 r 2 -1 r s 6 -21 1925 1807 i 6 -20 1926 1807 i -7 -12 1938 1803 i 9 -6 1934 1796 i 9 -7 1933 1796 i 4 1 1922 1806 i 6 2 1935 1802 i 7 2 1928 1785 i 1944 1790 1939 1789 b n 1945 1813 m -1 v 1 -1 r 1 1 r c s 4 -14 1947 1806 i 4 -13 1948 1806 i 3 1 1945 1805 i 1955 1793 1948 1791 b 4 -14 1958 1809 i 4 -13 1959 1809 i n 1960 1807 m 2 2 r 2 2 r 2 h 4 h 1 -1 r 3 -11 r s n 1966 1811 m 3 h 1 -2 r 3 -11 r s 4 1 1955 1808 i 7 1 1959 1795 i 1977 1800 1970 1798 b n 1983 1816 m -2 -1 r -2 v -1 -2 r 1 -2 r 1 -1 r 2 -1 r 2 -1 r 2 1 r 2 1 r 2 v 1 2 r -1 2 r -1 2 r -2 h -2 1 r c s n 1981 1815 m -3 v 1 -4 r 2 -1 r s n 1990 1808 m 3 v -1 4 r -2 1 r s n 1989 1816 m 1 v 2 2 r -1 v -2 -1 r s n 1982 1808 m -2 v -1 -2 r 1 -1 r 1 -2 r 3 h 5 2 r 3 -1 r 2 h s n 1982 1803 m 1 -1 r 3 h 5 2 r 3 -1 r 2 -1 r -1 v -3 v -3 -1 r -6 -2 r -3 h -2 2 r 1 v 2 v 3 2 r s 2 2007 1818 e 45 144 614 748 i 8 21 659 892 i 8 -12 634 897 i 8 -12 635 897 i n 636 895 m 1 2 r 2 3 r 2 1 r 3 1 r 2 -1 r 652 892 l s n 641 901 m 2 1 r 2 -2 r 651 891 l s 4 2 631 895 i 6 4 639 883 i 655 893 649 889 b n 656 912 m -2 -3 r -1 -3 r 1 -3 r 1 -2 r 3 -2 r 3 h 3 1 r 2 1 r 2 3 r 2 v -1 3 r -1 2 r -2 2 r -3 1 r -3 -1 r c s n 656 912 m -1 -2 r -1 -3 r 1 -3 r 1 -2 r 3 -2 r 3 -1 r 2 1 r s n 666 901 m 1 2 r 3 v -1 3 r -1 2 r -2 2 r -3 h -2 h s 8 -12 670 921 i 8 -12 670 922 i n 672 919 m 1 3 r 2 2 r 1 2 r 3 h 2 -1 r 7 -9 r s n 676 926 m 3 h 2 -1 r 687 915 l s 3 3 667 919 i 6 4 675 907 i 690 918 684 914 b 12 -18 684 940 i 12 -18 685 940 i n 691 932 m 3 v 1 2 r 2 1 r 3 1 r 3 -1 r 3 -2 r 1 -2 r 1 -3 r -1 -3 r -2 -2 r -1 -1 r -3 -1 r -3 1 r s n 694 938 m 2 h 3 h 3 -2 r 1 -2 r 1 -3 r -1 -3 r -1 -2 r s 685 940 682 938 b 12 -18 702 952 i 12 -17 703 952 i 4 2 699 950 i 718 936 712 932 b n 719 955 m -2 -3 r -1 -3 r 1 -3 r 1 -2 r 3 -2 r 3 h 3 1 r 2 1 r 2 2 r 3 v -1 3 r -1 2 r -3 2 r -2 1 r -4 -1 r c s n 719 955 m -1 -2 r -1 -3 r 1 -3 r 1 -2 r 3 -2 r 3 -1 r 2 1 r s n 729 944 m 1 2 r 3 v -1 3 r -1 2 r -2 1 r -3 1 r -3 h s n 743 967 m -1 -1 r 2 -1 r 2 v -1 1 r -2 h -3 h -2 -2 r -2 -2 r -1 -3 r 1 -3 r 1 -2 r 3 -2 r 3 -1 r 3 1 r 2 1 r 2 3 r 3 v s n 736 966 m -1 -2 r -1 -3 r 1 -3 r 1 -2 r 3 -2 r 3 h 2 h s 13 -18 744 981 i 12 -17 745 981 i -3 -14 758 981 i 10 -3 757 973 i 10 -4 756 973 i 3 2 742 979 i 5 4 755 979 i 6 4 754 961 i 768 971 763 968 b 19 -42 730 956 i 45 68 749 914 i 28 36 794 982 i 11 -9 825 1030 i 11 -9 826 1031 i n 828 1029 m 3 v 1 3 r 1 2 r 3 1 r 2 h 9 -8 r s n 830 1037 m 2 1 r 3 -1 r 8 -7 r s 3 3 823 1028 i 5 5 834 1019 i 846 1033 841 1027 b n 842 1051 m -1 -3 r -3 v 2 -3 r 1 -1 r 4 -1 r 2 h 3 1 r 2 2 r 1 3 r -1 3 r -1 3 r -2 1 r -3 1 r -3 h -3 -2 r c s n 842 1051 m -2 v -3 v 2 -3 r 1 -1 r 3 -2 r 3 1 r 2 h s n 855 1043 m 2 v 3 v -2 3 r -1 1 r -3 1 r -3 h -3 -1 r s 11 -9 853 1063 i 11 -9 853 1064 i n 856 1062 m 3 v 1 3 r 1 2 r 3 1 r 2 h 9 -7 r s n 858 1070 m 2 1 r 2 -1 r 9 -7 r s 2 3 851 1061 i 4 5 862 1052 i 874 1066 869 1060 b 16 -14 862 1085 i 17 -14 862 1086 i n 870 1079 m 3 v 1 3 r 1 1 r 3 2 r 3 h 3 -1 r 1 -1 r 2 -3 r -3 v -1 -3 r -1 -2 r -2 -1 r -3 h s n 872 1086 m 2 1 r 3 h 3 -1 r 2 -1 r 1 -3 r 1 -3 r -1 -2 r s 862 1086 860 1083 b 16 -14 876 1102 i 17 -14 876 1103 i 2 4 874 1099 i 895 1091 890 1086 b n 891 1109 m -1 -3 r -3 v 2 -3 r 1 -1 r 4 -1 r 2 h 3 2 r 2 1 r 1 3 r -1 3 r -1 3 r -2 1 r -3 1 r -3 h -2 -1 r c s n 891 1109 m -2 v -3 v 2 -3 r 1 -1 r 3 -1 r 3 h 2 1 r s n 904 1101 m 2 v 3 v -2 3 r -1 1 r -3 2 r -3 -1 r -2 h s n 911 1127 m -1 v 1 h 2 v -1 h -2 h -3 -1 r -2 -2 r -1 -3 r -3 v 2 -3 r 2 -1 r 3 -2 r 3 1 r 2 1 r 2 2 r 1 3 r 3 v s n 904 1125 m -3 v -2 v 2 -3 r 1 -2 r 3 -1 r 3 h 2 1 r s 16 -14 909 1141 i 16 -14 910 1142 i 1 -14 922 1145 i 10 923 1137 e 11 922 1136 e 3 3 907 1139 i 4 4 920 1143 i 5 5 923 1125 i 935 1138 931 1133 b 41 39 933 1150 i 45 48 974 1189 i 36 35 1019 1237 i 9 -11 1061 1283 i 10 -11 1061 1284 i n 1063 1282 m 1 2 r 1 3 r 2 2 r 3 1 r 2 -1 r 7 -9 r s n 1067 1289 m 2 h 2 -1 r 7 -9 r s 3 3 1058 1281 i 6 5 1067 1270 i 1082 1282 1076 1277 b n 1081 1300 m -2 -2 r -3 v 1 -3 r 2 -2 r 2 -2 r 3 h 3 1 r 2 2 r 2 2 r 3 v -1 3 r -2 2 r -2 2 r -3 h -3 -1 r c s n 1081 1300 m -1 -2 r -3 v 1 -3 r 1 -1 r 3 -2 r 3 h 2 h s n 1092 1291 m 1 2 r 3 v -1 3 r -1 1 r -3 2 r -3 h -2 h s 9 -11 1094 1311 i 9 -11 1095 1311 i n 1097 1309 m 3 v 2 3 r 1 1 r 3 1 r 2 -1 r 7 -8 r s n 1100 1316 m 2 1 r 2 -1 r 8 -9 r s 4 2 1091 1309 i 5 4 1101 1298 i 1115 1310 1109 1305 b 13 -17 1107 1331 i 14 -16 1107 1331 i n 1114 1324 m 2 v 1 3 r 2 1 r 3 1 r 3 h 2 -2 r 2 -2 r 1 -3 r -2 v -2 -3 r -2 -2 r -2 h -3 h s n 1117 1330 m 2 h 3 h 3 -2 r 1 -1 r 1 -3 r -3 v -1 -2 r s 1107 1331 1104 1329 b 14 -17 1123 1345 i 14 -16 1124 1345 i 3 2 1121 1343 i 1140 1331 1135 1326 b n 1140 1349 m -2 -3 r -3 v 1 -3 r 1 -1 r 3 -2 r 3 h 3 1 r 2 1 r 1 3 r 1 3 r -2 3 r -1 1 r -3 2 r -3 h -3 -1 r c s n 1140 1349 m -1 -2 r -3 v 1 -3 r 1 -2 r 3 -1 r 3 -1 r 2 1 r s n 1151 1339 m 2 v 1 3 r -1 3 r -2 2 r -3 1 r -2 1 r -3 -1 r s n 1162 1364 m -2 v 2 h 1 v -1 1 r -3 h -2 h -2 -2 r -2 -3 r -3 v 1 -3 r 1 -1 r 3 -2 r 3 h 3 1 r 1 1 r 2 3 r 3 v s n 1156 1362 m -1 -2 r -1 -3 r 2 -3 r 1 -2 r 3 -1 r 3 -1 r 2 1 r s 14 -16 1163 1377 i 13 -17 1164 1378 i -1 -14 1176 1379 i 10 -2 1176 1371 i 10 -2 1175 1370 i 3 3 1161 1375 i 5 4 1174 1377 i 6 4 1174 1359 i 1188 1370 1183 1366 b 11 8 1188 1382 i 45 30 1199 1390 i 45 28 1244 1420 i 45 23 1289 1448 i 1 1 1334 1471 i 8 -12 1342 1482 i 8 -12 1343 1482 i n 1345 1480 m 1 3 r 2 2 r 1 1 r 4 1 r 2 -1 r 5 -10 r s n 1349 1486 m 3 h 2 -1 r 6 -10 r s 3 2 1340 1480 i 6 4 1347 1468 i 1363 1478 1357 1474 b n 1365 1496 m -2 -3 r -3 v -3 v 1 -1 r 3 -3 r 3 h 3 h 2 1 r 2 3 r 3 v 3 v -1 2 r -3 2 r -3 1 r -3 -1 r c s n 1365 1496 m -1 -2 r -1 -3 r 1 -3 r 1 -2 r 3 -2 r 2 -1 r 3 h s n 1375 1484 m 1 2 r 3 v 3 v -1 2 r -3 2 r -3 1 r -2 h s 8 -12 1379 1504 i 8 -13 1380 1505 i n 1382 1502 m 1 3 r 2 2 r 1 1 r 4 1 r 2 -1 r 5 -10 r s n 1386 1508 m 3 1 r 2 -2 r 6 -9 r s 3 2 1377 1503 i 6 4 1384 1490 i 1400 1500 1394 1496 b 11 -18 1395 1522 i 11 -19 1396 1523 i n 1401 1514 m 1 3 r 1 2 r 2 1 r 3 h 3 h 2 -2 r 1 -2 r 1 -3 r -1 -3 r -2 -3 r -1 -1 r -3 h -3 1 r s n 1405 1520 m 2 h 3 -1 r 3 -2 r 1 -2 r -3 v -3 v -2 -2 r s 1396 1523 1392 1520 b 11 -18 1414 1533 i 12 -19 1414 1534 i 3 2 1411 1532 i 1428 1517 1422 1513 b n 1430 1535 m -2 -2 r -3 v -3 v 1 -2 r 3 -2 r 3 -1 r 3 1 r 2 1 r 2 2 r 3 v 3 v -1 2 r -3 2 r -3 1 r -3 -1 r c s n 1430 1535 m -1 -2 r -3 v -3 v 1 -2 r 3 -2 r 3 h 2 h s n 1440 1524 m 1 2 r 1 3 r -1 3 r -1 1 r -3 3 r -2 h -3 h s n 1455 1546 m -1 v 1 -1 r 2 v 1 v -3 h -2 h -3 -1 r -2 -3 r -1 -3 r 1 -3 r 1 -2 r 3 -2 r 2 h 4 h 1 1 r 2 3 r 1 3 r s n 1448 1546 m -1 -2 r -1 -3 r 1 -3 r 1 -2 r 2 -2 r 3 -1 r 3 h s 11 -19 1458 1560 i 12 -18 1458 1560 i -4 -14 1471 1559 i 9 -3 1470 1551 i 9 -4 1469 1551 i 3 2 1455 1558 i 6 3 1468 1558 i 6 3 1466 1540 i 1481 1549 1476 1546 b 31 15 1483 1561 i 45 19 1514 1576 i 45 13 1559 1595 i 39 18 1604 1608 i 5 -13 1651 1635 i 5 -14 1652 1636 i n 1653 1633 m 1 3 r 3 2 r 2 h 3 1 r 2 -2 r 4 -10 r s n 1659 1638 m 2 h 2 -1 r 4 -11 r s 4 2 1648 1634 i 6 3 1654 1621 i 1671 1628 1664 1625 b n 1676 1645 m -3 -2 r -1 -2 r -4 v 1 -1 r 2 -3 r 3 -1 r 3 h 2 1 r 3 2 r 1 3 r -1 3 r 2 v -2 2 r -3 1 r -3 h c s n 1676 1645 m -2 -1 r -1 -3 r -3 v 1 -2 r 2 -3 r 3 -1 r 2 h s n 1683 1633 m 2 2 r 1 2 r 4 v -1 1 r -2 3 r -3 1 r -2 h s 5 -14 1691 1652 i 5 -13 1692 1652 i n 1693 1649 m 1 3 r 3 2 r 2 1 r 3 h 2 -2 r 4 -10 r s n 1699 1655 m 2 h 2 -2 r 4 -10 r s 4 1 1688 1651 i 6 3 1694 1637 i 1711 1644 1704 1641 b 8 -20 1709 1667 i 8 -20 1710 1667 i n 1714 1658 m 1 2 r 2 2 r 2 1 r 3 h 3 -1 r 2 -2 r -2 v 1 -4 r -1 -2 r -3 -2 r -2 -1 r -2 h -3 1 r s n 1719 1663 m 2 h 3 -1 r 2 -3 r -2 v 1 -3 r -1 -3 r -2 -1 r s 1710 1667 1706 1666 b 8 -20 1729 1675 i 8 -20 1730 1675 i 4 1 1726 1674 i 1741 1657 1735 1654 b n 1746 1674 m -2 -2 r -1 -3 r -3 v 1 -2 r 2 -2 r 2 -1 r 4 h 2 1 r 2 2 r 1 2 r 4 v -1 2 r -2 2 r -3 1 r -3 h c s n 1746 1674 m -1 -2 r -1 -2 r -3 v 1 -2 r 2 -3 r 2 -1 r 3 h s n 1754 1662 m 1 1 r 1 3 r 3 v -1 2 r -2 3 r -3 1 r -2 h s n 1772 1681 m -1 v 1 h 1 v 1 v -3 1 r -2 h -3 -1 r -2 -2 r -1 -3 r -3 v 1 -2 r 2 -2 r 2 -2 r 4 1 r 2 h 2 3 r 1 2 r s n 1765 1682 m -1 -2 r -1 -2 r -4 v 1 -2 r 2 -2 r 2 -1 r 3 h s 8 -20 1777 1694 i 8 -20 1778 1695 i -6 -13 1790 1692 i 8 -5 1788 1684 i 8 -5 1787 1684 i 4 2 1774 1693 i 6 2 1787 1691 i 7 3 1782 1673 i 1798 1680 1793 1678 b 27 8 1802 1692 i 45 9 1829 1700 i 45 22 1874 1709 i 45 15 1919 1731 i 1 1964 1746 e 44 12 1965 1746 i 45 143 614 750 i 6 16 659 893 i 20 -8 658 924 i 20 -8 658 925 i n 657 921 m 4 12 r 2 2 r 2 1 r 2 h 2 -1 r 1 -1 r 1 -2 r -3 v -3 -8 r s n 661 933 m 2 1 r 1 1 r 3 h 1 -1 r 2 -1 r 1 -2 r -2 v s 2 7 677 913 i n 670 926 m 1 2 r 2 h 7 h 2 1 r 1 v 1 v s n 671 928 m 3 h 7 -1 r 1 1 r 1 2 r -1 1 r -1 1 r s n 678 939 m 5 12 r -2 1 r -2 -1 r -2 h -2 -2 r -1 -3 r -3 v 1 -3 r 3 -2 r 2 h 3 -1 r 3 2 r 2 2 r 2 v 1 3 r -1 3 r s n 682 950 m -3 1 r -2 h s n 674 946 m 1 -2 r 1 -3 r 2 -2 r 2 h 3 -1 r 3 2 r 2 1 r s n 682 960 m 1 h -1 v -1 h -1 1 r 3 v 1 3 r 2 2 r 1 1 r 3 h 6 -3 r 3 h 1 1 r s n 685 969 m 9 -4 r 2 h 2 2 r 1 v s n 687 968 m 1 -2 r -1 -6 r -1 -3 r 2 -2 r 2 h 2 h 2 2 r 1 3 r 2 v -1 3 r s n 687 960 m -2 v 1 -2 r 2 h 2 h 2 1 r s 20 -8 685 992 i 20 -8 685 993 i n 694 988 m -2 -1 r -2 -1 r -1 -2 r -4 v 1 -2 r 3 -2 r 2 -1 r 3 h 2 1 r 3 2 r 2 v 2 v -1 3 r s n 689 984 m -3 v 2 -2 r 2 -2 r 2 -1 r 3 h 3 1 r 2 1 r s 1 4 684 989 i 706 988 705 984 b 16 695 999 e 13 696 1000 e n 700 1010 m 711 999 l 3 -3 r 1 -3 r -2 v -1 v -1 -1 r -1 2 r 1 h s 3 6 694 997 i 701 1012 698 1007 b 20 -8 696 1021 i 20 -8 697 1022 i n 695 1018 m 5 11 r 2 3 r 1 h 2 1 r 2 -1 r 2 -2 r -1 v -4 v -3 -7 r s n 700 1029 m 1 2 r 2 h 2 1 r 2 -1 r 1 -2 r 1 -1 r -3 v s 3 6 715 1010 i n 708 1023 m 2 1 r 1 1 r 8 h 1 h 1 1 r -1 2 r s n 710 1024 m 2 h 7 h 2 h 1 2 r -2 2 r -1 h s n 718 1047 m -1 v 2 1 r -1 1 r -1 h -2 -1 r -2 -1 r -1 -3 r -3 v 1 -3 r 2 -2 r 2 -1 r 3 h 3 1 r 2 3 r 1 1 r 4 v -1 2 r s n 713 1043 m -2 v 1 -3 r 3 -2 r 1 -1 r 4 h 2 1 r 2 2 r s 15 1 718 1055 i 13 718 1056 e 11 -11 722 1067 i 2 6 717 1053 i 723 1069 721 1063 b 14 -5 725 1074 i 13 -5 726 1075 i n 731 1073 m -2 2 r -1 3 r 2 v 1 3 r 1 1 r 1 -1 r 1 -1 r -2 -1 r 2 v s 2 4 724 1071 i 740 1073 738 1066 b 12 28 737 1090 i 45 90 749 1118 i 18 31 794 1208 i 18 -12 808 1255 i 18 -12 808 1256 i n 806 1253 m 7 10 r 2 2 r 2 h 2 h 2 -1 r 1 -2 r -2 v -1 -3 r -4 -7 r s n 813 1263 m 2 1 r 1 h 2 h 2 -1 r 1 -2 r 1 -1 r -1 -3 r s 4 6 824 1241 i n 820 1255 m 2 1 r 1 h 8 -1 r 1 h 1 1 r 1 v s n 822 1256 m 2 h 7 -3 r 2 1 r 1 1 r -1 2 r -1 1 r s n 831 1266 m 6 10 r -1 1 r -3 1 r -1 -1 r -2 -1 r -2 -2 r -4 v -2 v 2 -3 r 2 -1 r 3 -1 r 3 1 r 2 2 r 2 1 r 4 v 2 v s n 837 1275 m -3 2 r -2 h s n 828 1274 m -3 v 1 -3 r 2 -2 r 1 -1 r 4 -1 r 2 1 r 2 1 r s n 839 1285 m -1 v -1 1 r 1 v 2 v 2 4 r 2 1 r 2 h 2 h 6 -4 r 2 h 2 h s n 844 1293 m 7 -5 r 3 h 2 1 r 1 v s n 845 1292 m 1 -1 r -3 -6 r -1 -3 r 1 -2 r 2 -1 r 2 -1 r 3 2 r 2 3 r 2 v -1 3 r s n 843 1285 m -2 v 1 -2 r 2 -2 r 2 h 2 1 r s 18 -12 848 1316 i 18 -12 848 1317 i n 856 1311 m -3 -1 r -2 -1 r -1 -2 r -1 -3 r 1 -3 r 2 -2 r 2 -1 r 3 -1 r 3 h 2 2 r 1 2 r 1 2 r -1 3 r s n 850 1307 m -2 v 1 -3 r 2 -2 r 1 -2 r 3 h 3 h 2 1 r s 2 3 846 1314 i 868 1308 866 1304 b 16 -3 859 1321 i 13 -3 860 1322 i n 866 1331 m 9 -13 r 2 -4 r 1 -3 r -1 -2 r -1 v -2 h 1 v 1 1 r s 4 5 858 1319 i 867 1333 864 1328 b 19 -12 864 1342 i 18 -12 865 1343 i n 863 1340 m 7 10 r 2 2 r 1 h 3 h 2 -1 r 1 -2 r -2 v -1 -3 r -4 -7 r s n 870 1350 m 2 1 r 1 h 2 h 2 -1 r 1 -2 r -1 v -3 v s 4 6 881 1328 i n 876 1342 m 2 1 r 2 h 8 -1 r 1 h 1 1 r -1 1 r s n 878 1343 m 3 h 7 -3 r 1 1 r 1 1 r -1 2 r -1 1 r s n 891 1364 m -1 v 2 h -1 1 r 1 v -3 -1 r -2 -1 r -2 -2 r -1 -4 r 1 -2 r 2 -3 r 2 -1 r 3 -1 r 3 1 r 2 2 r 1 1 r 1 4 r 2 v s n 885 1361 m -3 v -3 v 2 -2 r 2 -1 r 3 -1 r 3 1 r 2 1 r s 16 -3 892 1372 i 13 -3 893 1373 i 9 -13 899 1382 i 4 5 891 1370 i 900 1384 897 1379 b 12 -8 904 1389 i 12 -8 904 1390 i n 909 1386 m -2 3 r 3 v 2 v 2 3 r 1 h 1 -1 r -1 v -1 h 1 v s 2 4 902 1386 i 918 1385 914 1379 b 11 15 918 1402 i 45 59 929 1417 i 45 46 974 1476 i 7 7 1019 1522 i 14 -17 1027 1545 i 14 -17 1028 1546 i n 1025 1543 m 9 8 r 3 1 r 2 h 2 -1 r 1 -2 r 1 -2 r -1 v -2 -3 r -7 -5 r s n 1034 1551 m 3 h 1 h 2 -1 r 1 -1 r 1 -2 r -2 v -1 -2 r s 6 5 1038 1526 i n 1038 1541 m 3 1 r 1 -1 r 7 -3 r 2 h 2 v s n 1041 1542 m 2 -1 r 6 -5 r 1 h 2 1 r -1 3 r s n 1052 1548 m 10 8 r -1 2 r -2 1 r -2 h -2 -1 r -3 -2 r -1 -2 r -1 -3 r 2 -3 r 1 -2 r 3 -2 r 3 h 3 1 r 1 2 r 2 2 r 3 v s n 1061 1555 m -2 3 r -2 1 r s n 1052 1556 m -2 v -1 -3 r 1 -3 r 2 -1 r 2 -2 r 3 h 3 h s n 1066 1565 m -1 v -1 v -1 1 r 1 v 1 2 r 3 3 r 2 h 2 h 2 -1 r 5 -5 r 2 -1 r 1 h s n 1073 1570 m 6 -7 r 2 -1 r 2 1 r 1 h s n 1074 1569 m -2 v -4 -4 r -2 -3 r 1 -2 r 1 -2 r 2 -1 r 3 1 r 3 2 r 2 v 1 3 r s n 1070 1563 m -1 -2 r -3 v 2 -1 r 2 -1 r 2 h s 13 -17 1084 1591 i 13 -17 1085 1592 i n 1090 1583 m -3 h -2 h -1 -1 r -2 -3 r -3 v 1 -3 r 1 -2 r 3 -1 r 3 -1 r 3 1 r 1 2 r 1 2 r 1 3 r s n 1084 1582 m -1 -2 r -1 -3 r 2 -3 r 1 -2 r 3 -2 r 2 h 3 h s 4 3 1081 1589 i 1101 1577 1097 1574 b 14 -7 1096 1592 i 12 -7 1097 1593 i n 1106 1600 m 4 -15 r 1 -5 r -3 v -1 -2 r -1 h -1 h 1 v 1 h s 5 4 1095 1591 i 1108 1601 1103 1597 b 13 -17 1108 1611 i 13 -17 1109 1611 i n 1105 1609 m 10 7 r 3 2 r 1 -1 r 3 -1 r 1 -1 r -2 v -2 v -2 -3 r -6 -5 r s n 1115 1616 m 2 1 r 2 h 2 -1 r 1 -2 r 1 -2 r -1 -2 r -1 -2 r s 6 4 1119 1592 i n 1119 1606 m 2 1 r 2 h 7 -4 r 1 h 1 1 r 1 v s n 1121 1607 m 2 -1 r 7 -4 r 1 h 2 1 r -1 2 r -1 1 r s n 1140 1623 m -1 v 1 h 1 v 1 v -3 h -3 h -2 -2 r -2 -3 r -3 v 1 -3 r 1 -1 r 3 -2 r 3 -1 r 3 2 r 2 1 r 2 3 r 3 v s n 1133 1622 m -1 -2 r -3 v 1 -3 r 1 -2 r 3 -2 r 3 h 2 1 r s 14 -7 1143 1630 i 12 -6 1144 1631 i 4 -15 1153 1638 i 5 4 1142 1629 i 1155 1639 1150 1635 b 10 -11 1159 1643 i 9 -11 1160 1644 i n 1164 1639 m -1 3 r 3 v 1 2 r 3 2 r 1 h 1 -1 r -1 v -2 h 1 v s 3 3 1157 1641 i 1172 1635 1166 1630 b 22 20 1177 1651 i 45 32 1199 1671 i 45 26 1244 1703 i 24 18 1289 1729 i 10 -19 1317 1763 i 10 -20 1318 1764 i n 1314 1762 m 11 5 r 4 1 r 1 -1 r 2 -1 r 1 -2 r -2 v -1 -2 r -2 -2 r -7 -4 r s n 1325 1767 m 3 h 1 h 2 -2 r 1 -1 r -3 v -1 -1 r -1 -2 r s 7 4 1324 1742 i n 1327 1757 m 3 h 1 -1 r 6 -5 r 1 h 1 h 1 2 r s n 1330 1757 m 1 -2 r 5 -5 r 2 -1 r 2 1 r 3 v -1 1 r s n 1342 1761 m 11 6 r -1 1 r -1 2 r -2 h -2 h -3 -1 r -2 -3 r -1 -2 r -4 v 1 -1 r 3 -3 r 2 -1 r 4 1 r 2 1 r 2 2 r 1 3 r s n 1352 1766 m -1 3 r -2 1 r s n 1344 1769 m -1 -2 r -1 -3 r -3 v 1 -2 r 3 -2 r 2 -1 r 3 h s n 1359 1774 m -1 v -1 h 2 v 2 2 r 3 2 r 3 h 1 -1 r 2 -1 r 3 -7 r 2 -1 r 1 h s n 1367 1778 m 4 -8 r 2 -1 r 2 h 1 h s n 1368 1776 m -1 v -5 -4 r -3 -2 r -2 v 1 -2 r 2 -2 r 3 1 r 3 1 r 1 2 r 1 3 r s n 1363 1771 m -2 -2 r -2 v 1 -2 r 2 -1 r 2 h s 10 -19 1382 1796 i 10 -20 1383 1797 i n 1387 1787 m -3 1 r -2 h -2 -1 r -2 -2 r -1 -3 r -3 v 1 -2 r 2 -2 r 3 -1 r 3 h 2 1 r 2 2 r 1 3 r s n 1380 1787 m -1 -2 r -1 -3 r -3 v 1 -2 r 2 -2 r 3 -1 r 2 h s 4 2 1379 1795 i 1396 1779 1392 1777 b 12 -10 1395 1794 i 11 -9 1395 1795 i n 1406 1800 m 1 -16 r -4 v -1 -3 r -2 -2 r -1 h -1 h 2 v 2 -1 r s 5 3 1393 1793 i 1407 1801 1402 1798 b 9 -19 1410 1810 i 9 -20 1411 1811 i n 1407 1809 m 11 5 r 3 1 r 1 -1 r 2 -1 r 1 -2 r -2 v -1 v -2 -3 r -8 -4 r s n 1418 1814 m 2 h 2 h 1 -1 r 1 -2 r -3 v -1 v -1 -2 r s 6 3 1417 1790 i n 1420 1804 m 2 h 1 -1 r 6 -5 r 2 h 1 h 2 v s n 1422 1804 m 2 -2 r 5 -5 r 1 h 2 h 3 v 1 v s n 1444 1816 m -1 -1 r 2 -1 r 1 v 1 v -3 1 r -3 h -2 -1 r -3 -2 r -3 v -3 v 1 -2 r 2 -3 r 3 h 3 h 2 1 r 2 2 r 1 3 r s n 1437 1816 m -2 -2 r -1 -3 r 1 -3 r 1 -2 r 2 -2 r 3 -1 r 2 h s 12 -10 1449 1822 i 10 -8 1450 1822 i 1 -16 1460 1828 i 5 3 1447 1821 i 1461 1828 1456 1826 b 6 -13 1467 1831 i 6 -13 1468 1832 i n 1471 1826 m -1 3 r 1 3 r 1 2 r 3 1 r 2 h -1 v -1 v -2 h 1 2 r s 4 2 1464 1830 i 1477 1820 1471 1817 b 29 10 1485 1835 i 45 6 1514 1845 i 45 17 1559 1851 i 40 17 1604 1868 i 6 -21 1651 1900 i 6 -20 1652 1900 i n 1648 1899 m 11 4 r 4 h 1 -1 r 2 -2 r -2 v -2 v -1 -1 r -2 -2 r -8 -3 r s n 1659 1903 m 3 -1 r 1 h 2 -2 r -2 v -2 v -1 -1 r -1 -2 r s 7 3 1654 1878 i n 1660 1892 m 2 h 1 -1 r 5 -6 r 2 -1 r 1 1 r 1 v s n 1662 1892 m 1 -2 r 5 -6 r 1 -1 r 2 1 r 2 v 1 v s n 1675 1894 m 12 4 r -1 1 r -1 2 r -2 1 r -2 h -3 -1 r -2 -2 r -2 -2 r -4 v 1 -1 r 2 -3 r 2 -1 r 4 h 2 h 2 2 r 2 3 r s n 1686 1897 m -1 3 r -2 2 r s n 1678 1901 m -1 -1 r -2 -3 r -3 v 1 -2 r 2 -3 r 2 -1 r 3 h s n 1694 1904 m -1 v -1 h 1 v 1 v 2 2 r 4 1 r 2 -1 r 1 h 2 -2 r 2 -7 r 2 -1 r 1 -1 r s n 1702 1907 m 3 -9 r 2 -2 r 2 h 1 h s n 1703 1905 m -1 -2 r -5 -2 r -3 -2 r -3 v -2 v 2 -1 r 3 h 3 1 r 2 1 r 1 3 r s n 1697 1901 m -2 -2 r -2 v -2 v 2 -2 r 2 h s 7 -20 1720 1922 i 7 -20 1721 1922 i n 1723 1912 m -2 2 r -3 h -2 -1 r -2 -2 r -1 -2 r -1 -3 r 1 -2 r 2 -3 r 3 -1 r 3 h 2 h 1 2 r 2 2 r s n 1716 1913 m -1 -1 r -2 -3 r -3 v 1 -2 r 2 -3 r 3 -1 r 2 h s 4 1 1717 1921 i 1731 1903 1727 1902 b 10 -12 1732 1918 i 9 -11 1733 1919 i n 1744 1922 m -2 -16 r -4 v -2 -3 r -1 -1 r -1 -1 r -2 1 r 1 1 r 1 h s 6 2 1730 1918 i 1746 1923 1740 1921 b 6 -20 1750 1931 i 6 -21 1751 1932 i n 1747 1930 m 11 4 r 4 h 1 h 2 -2 r -2 v -2 v -1 -2 r -3 -2 r -7 -2 r s n 1758 1934 m 3 h 1 -1 r 2 -1 r -2 v -3 v -1 -1 r -2 -2 r s 7 2 1753 1910 i n 1759 1924 m 2 -1 r 1 -1 r 5 -5 r 2 -1 r 1 h 1 v s n 1761 1923 m 1 -1 r 5 -7 r 1 h 2 h 2 v 1 v s n 1784 1932 m -1 -2 r 2 h 1 v 1 v -3 1 r -2 1 r -3 -1 r -2 -2 r -2 -3 r -3 v 1 -2 r 2 -3 r 2 -1 r 4 h 2 1 r 2 1 r 2 3 r s n 1777 1933 m -1 -2 r -2 -3 r -3 v 1 -2 r 2 -2 r 2 -2 r 3 h s 10 -12 1790 1937 i 9 -10 1791 1937 i -2 -15 1802 1940 i 6 2 1788 1936 i 1804 1941 1798 1939 b 4 -14 1810 1943 i 4 -13 1811 1943 i n 1812 1937 m 4 v 2 2 r 1 2 r 3 1 r 2 -1 r -1 v -1 -1 r -1 h 1 2 r s 4 1 1807 1942 i 1818 1930 1811 1928 b 1 1828 1944 e 45 11 1829 1944 i 45 11 1874 1955 i 45 15 1919 1966 i 26 5 1964 1981 i 19 4 1990 1986 i end %%EndDocument @endspecial 80 2259 a Fa(Figure)16 b(1:)j(Comm)o(unications)d(rates)e(for)h (\\ping-p)q(ong")h(b)q(et)o(w)o(een)f(t)o(w)o(o)f(no)q(des)i(on)f(an)g(In)o (tel)h(ipsc/860.)963 2790 y(2)p eop %%Page: 3 3 bop 74 2133 a @beginspecial 72 @llx 72 @lly 504 @urx 504 @ury 4320 @rwi @setspecial %%BeginDocument: ring.ps .24 .24 scale /g0dict 40 dict def g0dict begin /m { moveto } bind def /a { rmoveto } bind def /l { lineto } bind def /v { 0 exch rlineto } bind def /h { 0 rlineto } bind def /s { stroke } bind def /n { newpath } bind def /r { rlineto } bind def /c { closepath } bind def /f { fill } bind def /p { copypage erasepage } def /g { setgray } bind def /b { newpath moveto lineto stroke } bind def /d { newpath moveto 0 exch rlineto stroke } bind def /e { newpath moveto 0 rlineto stroke } bind def /i { newpath moveto rlineto stroke } bind def /rs { dup stringwidth pop 0 exch sub 0 a show } bind def /cs { dup stringwidth pop 2 div 0 exch sub 0 a show } bind def /rshow { gsave currentpoint translate rotate 0 0 moveto show grestore } bind def /rrs { gsave currentpoint translate rotate 0 0 moveto dup stringwidth pop 0 exch sub 0 a show grestore } bind def /rcs { gsave currentpoint translate rotate 0 0 moveto dup stringwidth pop 2 div 0 exch sub 0 a show grestore } bind def 1440 569 569 e 36 569 569 d 18 713 569 d 36 857 569 d 18 1001 569 d 36 1145 569 d 18 1289 569 d 36 1433 569 d 18 1577 569 d 36 1721 569 d 18 1865 569 d 36 2009 569 d n 568 558 m -4 -2 r -2 -3 r 560 546 l -4 v 562 536 l 2 -4 r 4 -1 r 3 h 3 1 r 3 4 r 1 6 r 4 v -1 7 r -3 3 r -3 2 r c s n 568 558 m -3 -2 r -1 -1 r -1 -2 r 562 546 l -4 v 563 536 l 1 -3 r 1 -1 r 3 -1 r s n 571 531 m 2 1 r 1 1 r 2 3 r 1 6 r 4 v -1 7 r -2 2 r -1 1 r -2 2 r s n 814 553 m 2 1 r 4 4 r -27 v s -25 819 556 d 11 814 531 e n 843 558 m -4 -2 r -2 -3 r 835 546 l -4 v 837 536 l 2 -4 r 4 -1 r 3 h 3 1 r 3 4 r 1 6 r 4 v -1 7 r -3 3 r -3 2 r c s n 843 558 m -3 -2 r -1 -1 r -1 -2 r 837 546 l -4 v 838 536 l 1 -3 r 1 -1 r 3 -1 r s n 846 531 m 2 1 r 1 1 r 2 3 r 1 6 r 4 v -1 7 r -2 2 r -1 1 r -2 2 r s n 869 558 m -4 -2 r -3 -3 r 861 546 l -4 v 862 536 l 3 -4 r 4 -1 r 2 h 4 1 r 3 4 r 1 6 r 4 v -1 7 r -3 3 r -4 2 r c s n 869 558 m -3 -2 r -1 -1 r -1 -2 r 862 546 l -4 v 864 536 l 1 -3 r 1 -1 r 3 -1 r s n 871 531 m 3 1 r 1 1 r 1 3 r 2 6 r 4 v -2 7 r -1 2 r -1 1 r -3 2 r s n 894 558 m -3 -2 r -3 -3 r 887 546 l -4 v 888 536 l 3 -4 r 3 -1 r 3 h 4 1 r 2 4 r 2 6 r 4 v -2 7 r -2 3 r -4 2 r c s n 894 558 m -2 -2 r -1 -1 r -2 -2 r 888 546 l -4 v 889 536 l 2 -3 r 1 -1 r 2 -1 r s n 897 531 m 3 1 r 1 1 r 1 3 r 1 6 r 4 v -1 7 r -1 2 r -1 1 r -3 2 r s n 1099 553 m 1 -2 r -1 -1 r -2 1 r 2 v 2 2 r 1 1 r 4 2 r 5 h 4 -2 r 1 -1 r 1 -2 r -3 v -1 -2 r -4 -3 r -6 -3 r -3 -1 r -2 -2 r -2 -4 r -4 v s n 1109 558 m 3 -2 r 1 -1 r 1 -2 r -3 v -1 -2 r -4 -3 r -5 -3 r s n 1097 533 m 2 2 r 2 h 7 -3 r 4 h 2 1 r 1 2 r s n 1101 535 m 7 -4 r 5 h 1 1 r 1 3 r 2 v s n 1131 558 m -4 -2 r -3 -3 r 1123 546 l -4 v 1124 536 l 3 -4 r 4 -1 r 2 h 4 1 r 3 4 r 1 6 r 4 v -1 7 r -3 3 r -4 2 r c s n 1131 558 m -3 -2 r -1 -1 r -1 -2 r 1124 546 l -4 v 1126 536 l 1 -3 r 1 -1 r 3 -1 r s n 1133 531 m 3 1 r 1 1 r 2 3 r 1 6 r 4 v -1 7 r -2 2 r -1 1 r -3 2 r s n 1157 558 m -4 -2 r -3 -3 r 1149 546 l -4 v 1150 536 l 3 -4 r 4 -1 r 2 h 4 1 r 3 4 r 1 6 r 4 v -1 7 r -3 3 r -4 2 r c s n 1157 558 m -3 -2 r -1 -1 r -2 -2 r 1150 546 l -4 v 1151 536 l 2 -3 r 1 -1 r 3 -1 r s n 1159 531 m 3 1 r 1 1 r 1 3 r 2 6 r 4 v -2 7 r -1 2 r -1 1 r -3 2 r s n 1182 558 m -4 -2 r -2 -3 r 1175 546 l -4 v 1176 536 l 2 -4 r 4 -1 r 3 h 4 1 r 2 4 r 2 6 r 4 v -2 7 r -2 3 r -4 2 r c s n 1182 558 m -2 -2 r -2 -1 r -1 -2 r 1176 546 l -4 v 1177 536 l 1 -3 r 2 -1 r 2 -1 r s n 1185 531 m 2 1 r 2 1 r 1 3 r 1 6 r 4 v -1 7 r -1 2 r -2 1 r -2 2 r s n 1387 554 m 1 -1 r -1 -2 r -2 2 r 1 v 3 2 r 4 2 r 5 h 4 -2 r 1 -2 r -4 v -1 -2 r -4 -2 r -4 h s n 1397 558 m 2 -2 r 2 -2 r -4 v -2 -2 r -2 -2 r s n 1397 546 m 2 -1 r 3 -3 r 1 -2 r -4 v -1 -3 r -1 -1 r -4 -1 r -5 h -4 1 r -1 1 r -2 3 r 1 v 2 2 r 1 -2 r -1 -1 r s n 1401 544 m 1 -4 r -4 v -1 -3 r -2 -1 r -2 -1 r s n 1419 558 m -4 -2 r -3 -3 r 1411 546 l -4 v 1412 536 l 3 -4 r 4 -1 r 2 h 4 1 r 3 4 r 1 6 r 4 v -1 7 r -3 3 r -4 2 r c s n 1419 558 m -3 -2 r -1 -1 r -1 -2 r 1412 546 l -4 v 1414 536 l 1 -3 r 1 -1 r 3 -1 r s n 1421 531 m 3 1 r 1 1 r 1 3 r 2 6 r 4 v -2 7 r -1 2 r -1 1 r -3 2 r s n 1444 558 m -3 -2 r -3 -3 r 1437 546 l -4 v 1438 536 l 3 -4 r 3 -1 r 3 h 4 1 r 2 4 r 2 6 r 4 v -2 7 r -2 3 r -4 2 r c s n 1444 558 m -2 -2 r -1 -1 r -2 -2 r 1438 546 l -4 v 1439 536 l 2 -3 r 1 -1 r 2 -1 r s n 1447 531 m 3 1 r 1 1 r 1 3 r 1 6 r 4 v -1 7 r -1 2 r -1 1 r -3 2 r s n 1470 558 m -4 -2 r -2 -3 r 1462 546 l -4 v 1464 536 l 2 -4 r 4 -1 r 3 h 4 1 r 2 4 r 1 6 r 4 v -1 7 r -2 3 r -4 2 r c s n 1470 558 m -2 -2 r -2 -1 r -1 -2 r 1464 546 l -4 v 1465 536 l 1 -3 r 2 -1 r 2 -1 r s n 1473 531 m 2 1 r 2 1 r 1 3 r 1 6 r 4 v -1 7 r -1 2 r -2 1 r -2 2 r s -24 1685 555 d -27 1686 558 d n 1686 558 m 1672 539 l 20 h s 9 1681 531 e n 1707 558 m -4 -2 r -3 -3 r 1699 546 l -4 v 1700 536 l 3 -4 r 4 -1 r 2 h 4 1 r 3 4 r 1 6 r 4 v -1 7 r -3 3 r -4 2 r c s n 1707 558 m -3 -2 r -1 -1 r -2 -2 r 1700 546 l -4 v 1701 536 l 2 -3 r 1 -1 r 3 -1 r s n 1709 531 m 3 1 r 1 1 r 1 3 r 2 6 r 4 v -2 7 r -1 2 r -1 1 r -3 2 r s n 1732 558 m -4 -2 r -2 -3 r 1725 546 l -4 v 1726 536 l 2 -4 r 4 -1 r 3 h 4 1 r 2 4 r 2 6 r 4 v -2 7 r -2 3 r -4 2 r c s n 1732 558 m -2 -2 r -2 -1 r -1 -2 r 1726 546 l -4 v 1727 536 l 1 -3 r 2 -1 r 2 -1 r s n 1735 531 m 2 1 r 2 1 r 1 3 r 1 6 r 4 v -1 7 r -1 2 r -2 1 r -2 2 r s n 1758 558 m -4 -2 r -2 -3 r 1750 546 l -4 v 1752 536 l 2 -4 r 4 -1 r 3 h 3 1 r 3 4 r 1 6 r 4 v -1 7 r -3 3 r -3 2 r c s n 1758 558 m -3 -2 r -1 -1 r -1 -2 r 1752 546 l -4 v 1753 536 l 1 -3 r 1 -1 r 3 -1 r s n 1761 531 m 2 1 r 1 1 r 2 3 r 1 6 r 4 v -1 7 r -2 2 r -1 1 r -2 2 r s 1961 545 1964 558 b n 1961 545 m 3 3 r 3 1 r 4 h 4 -1 r 3 -3 r 1 -4 r -2 v -1 -4 r -3 -3 r -4 -1 r -4 h -3 1 r -2 1 r -1 3 r 1 v 1 2 r 2 -2 r -2 -1 r s n 1971 549 m 3 -1 r 2 -3 r 2 -4 r -2 v -2 -4 r -2 -3 r -3 -1 r s 12 1964 558 e n 1964 556 m 6 h 6 2 r s n 1994 558 m -3 -2 r -3 -3 r 1987 546 l -4 v 1988 536 l 3 -4 r 3 -1 r 3 h 4 1 r 2 4 r 2 6 r 4 v -2 7 r -2 3 r -4 2 r c s n 1994 558 m -2 -2 r -1 -1 r -2 -2 r 1988 546 l -4 v 1989 536 l 2 -3 r 1 -1 r 2 -1 r s n 1997 531 m 3 1 r 1 1 r 1 3 r 1 6 r 4 v -1 7 r -1 2 r -1 1 r -3 2 r s n 2020 558 m -4 -2 r -2 -3 r 2012 546 l -4 v 2014 536 l 2 -4 r 4 -1 r 3 h 4 1 r 2 4 r 1 6 r 4 v -1 7 r -2 3 r -4 2 r c s n 2020 558 m -2 -2 r -2 -1 r -1 -2 r 2014 546 l -4 v 2015 536 l 1 -3 r 2 -1 r 2 -1 r s n 2023 531 m 2 1 r 2 1 r 1 3 r 1 6 r 4 v -1 7 r -1 2 r -2 1 r -2 2 r s n 2046 558 m -4 -2 r -3 -3 r 2038 546 l -4 v 2039 536 l 3 -4 r 4 -1 r 2 h 4 1 r 3 4 r 1 6 r 4 v -1 7 r -3 3 r -4 2 r c s n 2046 558 m -3 -2 r -1 -1 r -1 -2 r 2039 546 l -4 v 2041 536 l 1 -3 r 1 -1 r 3 -1 r s n 2048 531 m 3 1 r 1 1 r 2 3 r 1 6 r 4 v -1 7 r -2 2 r -1 1 r -3 2 r s 1440 569 569 d 36 569 569 e 18 569 641 e 18 569 713 e 18 569 785 e 18 569 857 e 36 569 929 e 18 569 1001 e 18 569 1073 e 18 569 1145 e 18 569 1217 e 36 569 1289 e 18 569 1361 e 18 569 1433 e 18 569 1505 e 18 569 1577 e 36 569 1649 e 18 569 1721 e 18 569 1793 e 18 569 1865 e 18 569 1937 e 36 569 2009 e n 528 585 m -4 -2 r -2 -3 r 521 573 l -4 v 522 563 l 2 -4 r 4 -1 r 3 h 4 1 r 2 4 r 2 6 r 4 v -2 7 r -2 3 r -4 2 r c s n 528 585 m -2 -2 r -2 -1 r -1 -2 r 522 573 l -4 v 523 563 l 1 -3 r 2 -1 r 2 -1 r s n 531 558 m 2 1 r 2 1 r 1 3 r 1 6 r 4 v -1 7 r -1 2 r -2 1 r -2 2 r s 392 932 395 945 b n 392 932 m 3 2 r 3 2 r 4 h 4 -2 r 3 -2 r 1 -4 r -3 v -1 -4 r -3 -2 r -4 -1 r -4 h -3 1 r -2 1 r -1 3 r 1 v 1 1 r 2 -1 r -2 -1 r s n 402 936 m 3 -2 r 2 -2 r 2 -4 r -3 v -2 -4 r -2 -2 r -3 -1 r s 12 395 945 e n 395 943 m 6 h 6 2 r s n 425 945 m -3 -2 r -3 -4 r -1 -6 r -4 v 1 -6 r 3 -4 r 3 -1 r 3 h 4 1 r 2 4 r 2 6 r 4 v -2 6 r -2 4 r -4 2 r c s n 425 945 m -2 -2 r -1 -1 r -2 -3 r -1 -6 r -4 v 1 -6 r 2 -3 r 1 -1 r 2 -1 r s n 428 918 m 3 1 r 1 1 r 1 3 r 1 6 r 4 v -1 6 r -1 3 r -1 1 r -3 2 r s n 451 945 m -4 -2 r -2 -4 r -2 -6 r -4 v 2 -6 r 2 -4 r 4 -1 r 3 h 4 1 r 2 4 r 1 6 r 4 v -1 6 r -2 4 r -4 2 r c s n 451 945 m -2 -2 r -2 -1 r -1 -3 r -1 -6 r -4 v 1 -6 r 1 -3 r 2 -1 r 2 -1 r s n 454 918 m 2 1 r 2 1 r 1 3 r 1 6 r 4 v -1 6 r -1 3 r -2 1 r -2 2 r s n 477 945 m -4 -2 r -3 -4 r -1 -6 r -4 v 1 -6 r 3 -4 r 4 -1 r 2 h 4 1 r 3 4 r 1 6 r 4 v -1 6 r -3 4 r -4 2 r c s n 477 945 m -3 -2 r -1 -1 r -1 -3 r -2 -6 r -4 v 2 -6 r 1 -3 r 1 -1 r 3 -1 r s n 479 918 m 3 1 r 1 1 r 2 3 r 1 6 r 4 v -1 6 r -2 3 r -1 1 r -3 2 r s n 503 945 m -4 -2 r -3 -4 r -1 -6 r -4 v 1 -6 r 3 -4 r 4 -1 r 2 h 4 1 r 3 4 r 1 6 r 4 v -1 6 r -3 4 r -4 2 r c s n 503 945 m -3 -2 r -1 -1 r -2 -3 r -1 -6 r -4 v 1 -6 r 2 -3 r 1 -1 r 3 -1 r s n 505 918 m 3 1 r 1 1 r 1 3 r 2 6 r 4 v -2 6 r -1 3 r -1 1 r -3 2 r s n 528 945 m -4 -2 r -2 -4 r -1 -6 r -4 v 1 -6 r 2 -4 r 4 -1 r 3 h 4 1 r 2 4 r 2 6 r 4 v -2 6 r -2 4 r -4 2 r c s n 528 945 m -2 -2 r -2 -1 r -1 -3 r -1 -6 r -4 v 1 -6 r 1 -3 r 2 -1 r 2 -1 r s n 531 918 m 2 1 r 2 1 r 1 3 r 1 6 r 4 v -1 6 r -1 3 r -2 1 r -2 2 r s n 370 1299 m 3 2 r 4 3 r -27 v s -26 375 1303 d 12 370 1277 e n 400 1304 m 396 1303 l -3 -4 r -1 -6 r -4 v 1 -6 r 3 -4 r 4 -2 r 2 h 4 2 r 3 4 r 1 6 r 4 v -1 6 r -3 4 r -4 1 r c s n 400 1304 m -3 -1 r -1 -1 r -1 -3 r -2 -6 r -4 v 2 -6 r 1 -3 r 1 -1 r 3 -2 r s n 402 1277 m 3 2 r 1 1 r 1 3 r 2 6 r 4 v -2 6 r -1 3 r -1 1 r -3 1 r s n 425 1304 m -3 -1 r -3 -4 r -1 -6 r -4 v 1 -6 r 3 -4 r 3 -2 r 3 h 4 2 r 2 4 r 2 6 r 4 v -2 6 r -2 4 r -4 1 r c s n 425 1304 m -2 -1 r -1 -1 r -2 -3 r -1 -6 r -4 v 1 -6 r 2 -3 r 1 -1 r 2 -2 r s n 428 1277 m 3 2 r 1 1 r 1 3 r 1 6 r 4 v -1 6 r -1 3 r -1 1 r -3 1 r s n 451 1304 m -4 -1 r -2 -4 r -2 -6 r -4 v 2 -6 r 2 -4 r 4 -2 r 3 h 4 2 r 2 4 r 1 6 r 4 v -1 6 r -2 4 r -4 1 r c s n 451 1304 m -2 -1 r -2 -1 r -1 -3 r -1 -6 r -4 v 1 -6 r 1 -3 r 2 -1 r 2 -2 r s n 454 1277 m 2 2 r 2 1 r 1 3 r 1 6 r 4 v -1 6 r -1 3 r -2 1 r -2 1 r s n 477 1304 m -4 -1 r -3 -4 r -1 -6 r -4 v 1 -6 r 3 -4 r 4 -2 r 2 h 4 2 r 3 4 r 1 6 r 4 v -1 6 r -3 4 r -4 1 r c s n 477 1304 m -3 -1 r -1 -1 r -1 -3 r -2 -6 r -4 v 2 -6 r 1 -3 r 1 -1 r 3 -2 r s n 479 1277 m 3 2 r 1 1 r 2 3 r 1 6 r 4 v -1 6 r -2 3 r -1 1 r -3 1 r s n 503 1304 m -4 -1 r -3 -4 r -1 -6 r -4 v 1 -6 r 3 -4 r 4 -2 r 2 h 4 2 r 3 4 r 1 6 r 4 v -1 6 r -3 4 r -4 1 r c s n 503 1304 m -3 -1 r -1 -1 r -2 -3 r -1 -6 r -4 v 1 -6 r 2 -3 r 1 -1 r 3 -2 r s n 505 1277 m 3 2 r 1 1 r 1 3 r 2 6 r 4 v -2 6 r -1 3 r -1 1 r -3 1 r s n 528 1304 m -4 -1 r -2 -4 r -1 -6 r -4 v 1 -6 r 2 -4 r 4 -2 r 3 h 4 2 r 2 4 r 2 6 r 4 v -2 6 r -2 4 r -4 1 r c s n 528 1304 m -2 -1 r -2 -1 r -1 -3 r -1 -6 r -4 v 1 -6 r 1 -3 r 2 -1 r 2 -2 r s n 531 1277 m 2 2 r 2 1 r 1 3 r 1 6 r 4 v -1 6 r -1 3 r -2 1 r -2 1 r s n 370 1659 m 3 1 r 4 4 r -27 v s -26 375 1663 d 12 370 1637 e 392 1651 395 1664 b n 392 1651 m 3 3 r 3 1 r 4 h 4 -1 r 3 -3 r 1 -4 r -2 v -1 -4 r -3 -3 r -4 -1 r -4 h -3 1 r -2 2 r -1 2 r 2 v 1 1 r 2 -1 r -2 -2 r s n 402 1655 m 3 -1 r 2 -3 r 2 -4 r -2 v -2 -4 r -2 -3 r -3 -1 r s 12 395 1664 e n 395 1663 m 6 h 6 1 r s n 425 1664 m -3 -1 r -3 -4 r -1 -6 r -4 v 1 -7 r 3 -4 r 3 -1 r 3 h 4 1 r 2 4 r 2 7 r 4 v -2 6 r -2 4 r -4 1 r c s n 425 1664 m -2 -1 r -1 -1 r -2 -3 r -1 -6 r -4 v 1 -7 r 2 -2 r 1 -2 r 2 -1 r s n 428 1637 m 3 1 r 1 2 r 1 2 r 1 7 r 4 v -1 6 r -1 3 r -1 1 r -3 1 r s n 451 1664 m -4 -1 r -2 -4 r -2 -6 r -4 v 2 -7 r 2 -4 r 4 -1 r 3 h 4 1 r 2 4 r 1 7 r 4 v -1 6 r -2 4 r -4 1 r c s n 451 1664 m -2 -1 r -2 -1 r -1 -3 r -1 -6 r -4 v 1 -7 r 1 -2 r 2 -2 r 2 -1 r s n 454 1637 m 2 1 r 2 2 r 1 2 r 1 7 r 4 v -1 6 r -1 3 r -2 1 r -2 1 r s n 477 1664 m -4 -1 r -3 -4 r -1 -6 r -4 v 1 -7 r 3 -4 r 4 -1 r 2 h 4 1 r 3 4 r 1 7 r 4 v -1 6 r -3 4 r -4 1 r c s n 477 1664 m -3 -1 r -1 -1 r -1 -3 r -2 -6 r -4 v 2 -7 r 1 -2 r 1 -2 r 3 -1 r s n 479 1637 m 3 1 r 1 2 r 2 2 r 1 7 r 4 v -1 6 r -2 3 r -1 1 r -3 1 r s n 503 1664 m -4 -1 r -3 -4 r -1 -6 r -4 v 1 -7 r 3 -4 r 4 -1 r 2 h 4 1 r 3 4 r 1 7 r 4 v -1 6 r -3 4 r -4 1 r c s n 503 1664 m -3 -1 r -1 -1 r -2 -3 r -1 -6 r -4 v 1 -7 r 2 -2 r 1 -2 r 3 -1 r s n 505 1637 m 3 1 r 1 2 r 1 2 r 2 7 r 4 v -2 6 r -1 3 r -1 1 r -3 1 r s n 528 1664 m -4 -1 r -2 -4 r -1 -6 r -4 v 1 -7 r 2 -4 r 4 -1 r 3 h 4 1 r 2 4 r 2 7 r 4 v -2 6 r -2 4 r -4 1 r c s n 528 1664 m -2 -1 r -2 -1 r -1 -3 r -1 -6 r -4 v 1 -7 r 1 -2 r 2 -2 r 2 -1 r s n 531 1637 m 2 1 r 2 2 r 1 2 r 1 7 r 4 v -1 6 r -1 3 r -2 1 r -2 1 r s n 368 2019 m 1 -1 r -1 -2 r -2 2 r 1 v 2 2 r 1 2 r 4 1 r 5 h 4 -1 r 1 -2 r 1 -2 r -3 v -1 -2 r 379 2011 l 373 2009 l -3 -2 r -2 -2 r -2 -4 r -4 v s n 378 2024 m 2 -1 r 2 -2 r 1 -2 r -3 v -1 -2 r 378 2011 l 373 2009 l s n 366 2000 m 2 1 r 2 h 7 -3 r 3 h 3 2 r 1 1 r s n 370 2001 m 7 -4 r 5 h 1 1 r 1 3 r 2 v s n 400 2024 m 396 2023 l -3 -4 r -1 -7 r -3 v 1 -7 r 3 -4 r 4 -1 r 2 h 4 1 r 3 4 r 1 7 r 3 v -1 7 r -3 4 r -4 1 r c s n 400 2024 m -3 -1 r -1 -2 r -1 -2 r -2 -7 r -3 v 2 -7 r 1 -2 r 1 -2 r 3 -1 r s n 402 1997 m 3 1 r 1 2 r 1 2 r 2 7 r 3 v -2 7 r -1 2 r -1 2 r -3 1 r s n 425 2024 m -3 -1 r -3 -4 r -1 -7 r -3 v 1 -7 r 3 -4 r 3 -1 r 3 h 4 1 r 2 4 r 2 7 r 3 v -2 7 r -2 4 r -4 1 r c s n 425 2024 m -2 -1 r -1 -2 r -2 -2 r -1 -7 r -3 v 1 -7 r 2 -2 r 1 -2 r 2 -1 r s n 428 1997 m 3 1 r 1 2 r 1 2 r 1 7 r 3 v -1 7 r -1 2 r -1 2 r -3 1 r s n 451 2024 m -4 -1 r -2 -4 r -2 -7 r -3 v 2 -7 r 2 -4 r 4 -1 r 3 h 4 1 r 2 4 r 1 7 r 3 v -1 7 r -2 4 r -4 1 r c s n 451 2024 m -2 -1 r -2 -2 r -1 -2 r -1 -7 r -3 v 1 -7 r 1 -2 r 2 -2 r 2 -1 r s n 454 1997 m 2 1 r 2 2 r 1 2 r 1 7 r 3 v -1 7 r -1 2 r -2 2 r -2 1 r s n 477 2024 m -4 -1 r -3 -4 r -1 -7 r -3 v 1 -7 r 3 -4 r 4 -1 r 2 h 4 1 r 3 4 r 1 7 r 3 v -1 7 r -3 4 r -4 1 r c s n 477 2024 m -3 -1 r -1 -2 r -1 -2 r -2 -7 r -3 v 2 -7 r 1 -2 r 1 -2 r 3 -1 r s n 479 1997 m 3 1 r 1 2 r 2 2 r 1 7 r 3 v -1 7 r -2 2 r -1 2 r -3 1 r s n 503 2024 m -4 -1 r -3 -4 r -1 -7 r -3 v 1 -7 r 3 -4 r 4 -1 r 2 h 4 1 r 3 4 r 1 7 r 3 v -1 7 r -3 4 r -4 1 r c s n 503 2024 m -3 -1 r -1 -2 r -2 -2 r -1 -7 r -3 v 1 -7 r 2 -2 r 1 -2 r 3 -1 r s n 505 1997 m 3 1 r 1 2 r 1 2 r 2 7 r 3 v -2 7 r -1 2 r -1 2 r -3 1 r s n 528 2024 m -4 -1 r -2 -4 r -1 -7 r -3 v 1 -7 r 2 -4 r 4 -1 r 3 h 4 1 r 2 4 r 2 7 r 3 v -2 7 r -2 4 r -4 1 r c s n 528 2024 m -2 -1 r -2 -2 r -1 -2 r -1 -7 r -3 v 1 -7 r 1 -2 r 2 -2 r 2 -1 r s n 531 1997 m 2 1 r 2 2 r 1 2 r 1 7 r 3 v -1 7 r -1 2 r -2 2 r -2 1 r s 1440 569 2009 e -36 569 2009 d -18 713 2009 d -36 857 2009 d -18 1001 2009 d -36 1145 2009 d -18 1289 2009 d -36 1433 2009 d -18 1577 2009 d -36 1721 2009 d -18 1865 2009 d -36 2009 2009 d 1440 2009 569 d -36 2009 569 e -18 2009 641 e -18 2009 713 e -18 2009 785 e -18 2009 857 e -36 2009 929 e -18 2009 1001 e -18 2009 1073 e -18 2009 1145 e -18 2009 1217 e -36 2009 1289 e -18 2009 1361 e -18 2009 1433 e -18 2009 1505 e -18 2009 1577 e -36 2009 1649 e -18 2009 1721 e -18 2009 1793 e -18 2009 1865 e -18 2009 1937 e -36 2009 2009 e -27 1139 510 d -27 1141 510 d 6 1135 510 e 9 1135 483 e n 1152 494 m 16 h 2 v -2 3 r -1 1 r -3 1 r -3 h -4 -1 r -3 -3 r -1 -3 r -3 v 1 -4 r 3 -2 r 4 -2 r 2 h 4 2 r 3 2 r s n 1166 494 m 3 v -1 3 r s n 1159 501 m -3 -1 r -3 -3 r -1 -3 r -3 v 1 -4 r 3 -2 r 3 -2 r s -18 1178 501 d -18 1179 501 d n 1179 497 m 3 3 r 4 1 r 2 h 4 -1 r 1 -3 r -14 v s n 1188 501 m 3 -1 r 1 -3 r -14 v s 5 1174 501 e 9 1174 483 e 9 1188 483 e n 1210 501 m -3 -1 r -1 -1 r -1 -3 r -2 v 1 -3 r 1 -1 r 3 -2 r 2 h 3 2 r 1 1 r 2 3 r 2 v -2 3 r -1 1 r -3 1 r c s n 1207 500 m -1 -3 r -5 v 1 -2 r s n 1215 490 m 1 2 r 5 v -1 3 r s n 1216 499 m 2 1 r 2 1 r -1 v -2 h s n 1206 491 m -1 -1 r -2 -3 r -1 v 2 -3 r 4 -1 r 6 h 4 -1 r 1 -2 r s n 1203 486 m 2 -1 r 4 -2 r 6 h 4 -1 r 1 -3 r -1 v -1 -2 r -4 -2 r -8 h -4 2 r -1 2 r 1 v 1 3 r 4 1 r s n 1230 510 m -22 v 2 -3 r 2 -2 r 3 h 2 2 r 2 2 r s n 1232 510 m -22 v 1 -3 r 1 -2 r s 10 1227 501 e -27 1250 510 d -27 1251 510 d n 1251 497 m 3 3 r 3 1 r 3 h 4 -1 r 1 -3 r -14 v s n 1260 501 m 3 -1 r 1 -3 r -14 v s 5 1246 510 e 9 1246 483 e 9 1260 483 e n 1306 515 m -2 -2 r -3 -4 r -2 -5 r 1297 497 l -5 v 1299 486 l 1301 481 l 3 -4 r 2 -3 r s n 1304 513 m -3 -5 r -1 -4 r 1299 497 l -5 v 1300 486 l 1 -4 r 1304 477 l s -27 1317 510 d -27 1318 510 d n 1318 497 m 2 3 r 3 1 r 3 h 3 -1 r 3 -3 r 1 -3 r -3 v -1 -4 r -3 -2 r -3 -2 r -3 h -3 2 r -2 2 r s n 1326 501 m 2 -1 r 3 -3 r 1 -3 r -3 v -1 -4 r -3 -2 r -2 -2 r s 5 1313 510 e 8 -18 1342 501 i 6 -15 1344 501 i n 1358 501 m 1350 483 l 1347 478 l -2 -2 r -3 -2 r -1 h -1 2 r 1 1 r 1 -1 r s 7 1340 501 e 7 1353 501 e n 1368 510 m -22 v 1 -3 r 3 -2 r 2 h 3 2 r 1 2 r s n 1369 510 m -22 v 2 -3 r 1 -2 r s 10 1364 501 e n 1386 494 m 15 h 2 v -1 3 r -1 1 r -3 1 r -4 h -3 -1 r -3 -3 r -1 -3 r -3 v 1 -4 r 3 -2 r 3 -2 r 3 h 4 2 r 2 2 r s n 1400 494 m 3 v -1 3 r s n 1392 501 m -2 -1 r -3 -3 r -1 -3 r -3 v 1 -4 r 3 -2 r 2 -2 r s n 1422 499 m 1 2 r -5 v -1 3 r -1 1 r -3 1 r -5 h -3 -1 r -1 -1 r -3 v 1 -1 r 3 -1 r 6 -3 r 3 -1 r 1 -2 r s n 1409 497 m 1 -1 r 3 -1 r 6 -3 r 3 -1 r 1 -1 r -4 v -1 -1 r -3 -2 r -5 h -2 2 r -2 1 r -1 2 r -5 v 1 3 r s n 1431 515 m 3 -2 r 2 -4 r 3 -5 r 1440 497 l -5 v 1439 486 l 1436 481 l -2 -4 r -3 -3 r s n 1434 513 m 2 -5 r 1 -4 r 1439 497 l -5 v 1437 486 l -1 -4 r 1434 477 l s -36 645 2065 d -36 646 2065 d n 640 2065 m 20 h 5 -2 r 2 -1 r 2 -4 r -3 v -2 -4 r -2 -1 r -5 -2 r -14 h s n 660 2065 m 4 -2 r 1 -1 r 2 -4 r -3 v -2 -4 r -1 -1 r -4 -2 r s 12 640 2029 e n 655 2048 m 3 -2 r 2 -1 r 5 -12 r 2 -2 r 2 h 1 2 r s n 658 2046 m 2 -3 r 4 -12 r 1 -2 r 4 h 1 4 r 1 v s n 682 2065 m -1 -2 r 1 -1 r 2 1 r c s -24 682 2053 d -24 684 2053 d 7 677 2053 e 12 677 2029 e -24 701 2053 d -24 703 2053 d n 703 2048 m 3 3 r 6 2 r 3 h 5 -2 r 2 -3 r -19 v s n 715 2053 m 3 -2 r 2 -3 r -19 v s 7 696 2053 e 12 696 2029 e 12 715 2029 e n 744 2053 m -3 -2 r -2 -1 r -2 -4 r -3 v 2 -4 r 2 -1 r 3 -2 r 4 h 3 2 r 2 1 r 1 4 r 3 v -1 4 r -2 1 r -3 2 r c s n 741 2051 m -2 -3 r -7 v 2 -3 r s n 751 2038 m 2 3 r 7 v -2 3 r s n 753 2050 m 1 1 r 4 2 r -2 v -4 h s n 739 2039 m -2 -1 r -1 -4 r -1 v 1 -4 r 5 -2 r 9 h 5 -1 r 2 -2 r s n 736 2033 m 1 -2 r 5 -2 r 9 h 5 -2 r 2 -3 r -2 v -2 -3 r -5 -2 r -10 h -5 2 r -2 3 r 2 v 2 3 r 5 2 r s n 819 2060 m 2 -5 r 10 v -2 -5 r -3 3 r -5 2 r -3 h -6 -2 r -3 -3 r -2 -3 r -1 -6 r -8 v 1 -5 r 2 -4 r 3 -3 r 6 -2 r 3 h 5 2 r 3 3 r 2 4 r s n 808 2065 m -4 -2 r -3 -3 r -2 -3 r -2 -6 r -8 v 2 -5 r 2 -4 r 3 -3 r 4 -2 r s n 842 2053 m -5 -2 r -4 -3 r -2 -5 r -4 v 2 -5 r 4 -3 r 5 -2 r 3 h 5 2 r 4 3 r 1 5 r 4 v -1 5 r -4 3 r -5 2 r c s n 842 2053 m -4 -2 r -3 -3 r -2 -5 r -4 v 2 -5 r 3 -3 r 4 -2 r s n 845 2029 m 4 2 r 3 3 r 2 5 r 4 v -2 5 r -3 3 r -4 2 r s -24 869 2053 d -24 871 2053 d n 871 2048 m 3 3 r 5 2 r 4 h 5 -2 r 2 -3 r -19 v s n 883 2053 m 3 -2 r 2 -3 r -19 v s n 890 2048 m 3 3 r 5 2 r 4 h 5 -2 r 2 -3 r -19 v s n 902 2053 m 3 -2 r 2 -3 r -19 v s 7 864 2053 e 12 864 2029 e 12 883 2029 e 12 902 2029 e -24 926 2053 d -24 927 2053 d n 927 2048 m 4 3 r 5 2 r 3 h 6 -2 r 1 -3 r -19 v s n 939 2053 m 4 -2 r 2 -3 r -19 v s n 946 2048 m 4 3 r 5 2 r 3 h 5 -2 r 2 -3 r -19 v s n 958 2053 m 4 -2 r 1 -3 r -19 v s 6 921 2053 e 12 921 2029 e 12 939 2029 e 12 958 2029 e n 982 2053 m -19 v 2 -3 r 5 -2 r 4 h 5 2 r 3 3 r s n 984 2053 m -19 v 2 -3 r 3 -2 r s -24 1001 2053 d -24 1003 2053 d 7 977 2053 e 7 996 2053 e 7 1001 2029 e -24 1020 2053 d -24 1022 2053 d n 1022 2048 m 3 3 r 5 2 r 4 h 5 -2 r 2 -3 r -19 v s n 1034 2053 m 3 -2 r 2 -3 r -19 v s 7 1015 2053 e 12 1015 2029 e 12 1034 2029 e n 1058 2065 m -2 -2 r 2 -1 r 1 1 r c s -24 1058 2053 d -24 1059 2053 d 6 1053 2053 e 12 1053 2029 e n 1094 2048 m -2 -2 r 2 -1 r 1 1 r 2 v -3 3 r -4 2 r -5 h -5 -2 r -3 -3 r -2 -5 r -4 v 2 -5 r 3 -3 r 5 -2 r 4 h 5 2 r 3 3 r s n 1083 2053 m -3 -2 r -4 -3 r -1 -5 r -4 v 1 -5 r 4 -3 r 3 -2 r s n 1109 2050 m -2 v -2 h 2 v 2 1 r 3 2 r 7 h 4 -2 r 1 -1 r 2 -4 r -12 v 2 -3 r 2 -2 r s n 1124 2050 m -16 v 2 -3 r 4 -2 r 1 h s n 1124 2046 m -1 -1 r -11 -2 r -5 -2 r -1 -3 r -4 v 1 -3 r 5 -2 r 6 h 3 2 r 3 3 r s n 1112 2043 m -3 -2 r -2 -3 r -4 v 2 -3 r 3 -2 r s n 1143 2065 m -29 v 2 -5 r 3 -2 r 4 h 3 2 r 2 3 r s n 1145 2065 m -29 v 2 -5 r 1 -2 r s 14 1138 2053 e n 1169 2065 m -2 -2 r 2 -1 r 2 1 r c s -24 1169 2053 d -24 1171 2053 d 7 1164 2053 e 12 1164 2029 e n 1195 2053 m -5 -2 r -4 -3 r -2 -5 r -4 v 2 -5 r 4 -3 r 5 -2 r 3 h 5 2 r 4 3 r 1 5 r 4 v -1 5 r -4 3 r -5 2 r c s n 1195 2053 m -4 -2 r -3 -3 r -2 -5 r -4 v 2 -5 r 3 -3 r 4 -2 r s n 1198 2029 m 4 2 r 3 3 r 2 5 r 4 v -2 5 r -3 3 r -4 2 r s -24 1222 2053 d -24 1224 2053 d n 1224 2048 m 3 3 r 5 2 r 4 h 5 -2 r 2 -3 r -19 v s n 1236 2053 m 3 -2 r 2 -3 r -19 v s 7 1217 2053 e 12 1217 2029 e 12 1236 2029 e n 1274 2050 m 1 3 r -7 v -1 4 r -2 1 r -4 2 r -6 h -4 -2 r -2 -1 r -4 v 2 -1 r 4 -2 r 8 -4 r 4 -1 r 1 -2 r s n 1256 2048 m 2 -2 r 4 -1 r 8 -4 r 4 -2 r 1 -1 r -5 v -1 -2 r -4 -2 r -7 h -3 2 r -2 2 r -2 3 r -7 v 2 4 r s -36 1316 2065 d -36 1318 2065 d n 1311 2065 m 21 h 5 -2 r 2 -1 r 1 -4 r -5 v -1 -3 r -2 -2 r -5 -2 r -14 h s n 1332 2065 m 3 -2 r 2 -1 r 2 -4 r -5 v -2 -3 r -2 -2 r -3 -2 r s 12 1311 2029 e n 1352 2043 m 21 h 3 v -2 4 r -2 1 r -3 2 r -5 h -5 -2 r -4 -3 r -1 -5 r -4 v 1 -5 r 4 -3 r 5 -2 r 3 h 5 2 r 4 3 r s n 1371 2043 m 5 v -2 3 r s n 1361 2053 m -4 -2 r -3 -3 r -2 -5 r -4 v 2 -5 r 3 -3 r 4 -2 r s -24 1387 2053 d -24 1388 2053 d n 1388 2043 m 2 5 r 3 3 r 4 2 r 5 h 2 -2 r -1 v -2 -2 r -2 2 r 2 1 r s 7 1381 2053 e 12 1381 2029 e n 1424 2063 m -1 -1 r 1 -2 r 2 2 r 1 v -2 2 r -3 h -4 -2 r -1 -3 r -31 v s n 1421 2065 m -2 -2 r -2 -3 r -31 v s 13 1411 2053 e 12 1411 2029 e n 1445 2053 m -5 -2 r -4 -3 r -1 -5 r -4 v 1 -5 r 4 -3 r 5 -2 r 3 h 5 2 r 4 3 r 2 5 r 4 v -2 5 r -4 3 r -5 2 r c s n 1445 2053 m -4 -2 r -3 -3 r -2 -5 r -4 v 2 -5 r 3 -3 r 4 -2 r s n 1448 2029 m 4 2 r 3 3 r 2 5 r 4 v -2 5 r -3 3 r -4 2 r s -24 1472 2053 d -24 1474 2053 d n 1474 2043 m 2 5 r 3 3 r 4 2 r 5 h 1 -2 r -1 v -1 -2 r -2 2 r 2 1 r s 7 1467 2053 e 12 1467 2029 e -24 1501 2053 d -24 1503 2053 d n 1503 2048 m 4 3 r 5 2 r 3 h 5 -2 r 2 -3 r -19 v s n 1515 2053 m 4 -2 r 1 -3 r -19 v s n 1522 2048 m 3 3 r 6 2 r 3 h 5 -2 r 2 -3 r -19 v s n 1534 2053 m 3 -2 r 2 -3 r -19 v s 7 1496 2053 e 12 1496 2029 e 12 1515 2029 e 12 1534 2029 e n 1558 2050 m -2 v -2 h 2 v 2 1 r 3 2 r 7 h 4 -2 r 1 -1 r 2 -4 r -12 v 2 -3 r 2 -2 r s n 1573 2050 m -16 v 2 -3 r 4 -2 r 1 h s n 1573 2046 m -1 -1 r -11 -2 r -5 -2 r -1 -3 r -4 v 1 -3 r 5 -2 r 6 h 3 2 r 3 3 r s n 1561 2043 m -3 -2 r -2 -3 r -4 v 2 -3 r 3 -2 r s -24 1592 2053 d -24 1594 2053 d n 1594 2048 m 3 3 r 5 2 r 4 h 5 -2 r 2 -3 r -19 v s n 1606 2053 m 3 -2 r 2 -3 r -19 v s 7 1587 2053 e 12 1587 2029 e 12 1606 2029 e n 1647 2048 m -2 -2 r 2 -1 r 2 1 r 2 v -4 3 r -3 2 r -5 h -5 -2 r -4 -3 r -2 -5 r -4 v 2 -5 r 4 -3 r 5 -2 r 3 h 5 2 r 4 3 r s n 1637 2053 m -4 -2 r -3 -3 r -2 -5 r -4 v 2 -5 r 3 -3 r 4 -2 r s n 1661 2043 m 20 h 3 v -1 4 r -2 1 r -4 2 r -5 h -5 -2 r -3 -3 r -2 -5 r -4 v 2 -5 r 3 -3 r 5 -2 r 4 h 5 2 r 3 3 r s n 1680 2043 m 5 v -2 3 r s n 1669 2053 m -3 -2 r -4 -3 r -1 -5 r -4 v 1 -5 r 4 -3 r 3 -2 r s n 1733 2072 m -4 -3 r -3 -6 r -4 -6 r -1 -9 r -7 v 1 -8 r 4 -7 r 3 -5 r 4 -4 r s n 1729 2069 m -3 -7 r -2 -5 r -2 -9 r -7 v 2 -8 r 2 -6 r 3 -6 r s -24 1746 2053 d -24 1748 2053 d n 1748 2048 m 4 3 r 5 2 r 3 h 5 -2 r 2 -3 r -19 v s n 1760 2053 m 4 -2 r 1 -3 r -19 v s 7 1741 2053 e 12 1741 2029 e 12 1760 2029 e -36 1784 2053 d -36 1786 2053 d n 1786 2048 m 3 3 r 4 2 r 3 h 5 -2 r 4 -3 r 1 -5 r -4 v -1 -5 r -4 -3 r -5 -2 r -3 h -4 2 r -3 3 r s n 1796 2053 m 4 -2 r 3 -3 r 2 -5 r -4 v -2 -5 r -3 -3 r -4 -2 r s 7 1779 2053 e 12 1779 2017 e 31 1846 2050 e 31 1846 2039 e -33 1904 2062 d -36 1906 2065 d n 1906 2065 m 1887 2039 l 27 h s 12 1899 2029 e n 1923 2072 m 3 -3 r 4 -6 r 3 -6 r 2 -9 r -7 v -2 -8 r -3 -7 r -4 -5 r -3 -4 r s n 1926 2069 m 4 -7 r 1 -5 r 2 -9 r -7 v -2 -8 r -1 -6 r -4 -6 r s 27 320 1110 e 27 320 1112 e n 320 1106 m 16 v 1 4 r 2 1 r 2 1 r 3 h 2 -1 r 2 -1 r 1 -4 r -10 v s n 320 1122 m 1 2 r 2 2 r 2 1 r 3 h 2 -1 r 2 -2 r 1 -2 r s 9 347 1106 d n 333 1118 m 1 3 r 1 1 r 9 4 r 2 1 r 1 v -2 2 r s n 334 1121 m 3 1 r 9 2 r 1 2 r 2 v -3 2 r -1 h s n 332 1139 m 1 h -2 v -1 h -2 2 r -1 2 r 5 v 1 3 r 2 1 r 2 1 r 9 h 3 2 r 1 1 r s n 332 1150 m 11 h 3 1 r 1 3 r 1 v s n 334 1150 m 1 -1 r 2 -8 r 1 -4 r 3 -1 r 2 h 3 1 r 1 4 r 4 v -1 3 r -3 2 r s n 337 1141 m 1 -2 r 3 -2 r 2 h 3 2 r 1 2 r s n 320 1164 m 22 h 4 2 r 1 2 r 3 v -1 2 r -3 2 r s n 320 1166 m 22 h 4 1 r 1 1 r s 11 329 1160 d n 337 1182 m 16 v -3 h -2 -2 r -2 -1 r -1 -2 r -4 v 1 -4 r 3 -3 r 4 -1 r 2 h 4 1 r 3 3 r 1 4 r 2 v -1 4 r -3 3 r s n 337 1196 m -4 h -3 -1 r s n 329 1189 m 1 -3 r 3 -2 r 4 -2 r 2 h 4 2 r 3 2 r 1 3 r s n 315 1236 m 2 -2 r 4 -3 r 5 -2 r 7 -2 r 5 h 6 2 r 6 2 r 3 3 r 3 2 r s n 317 1234 m 6 -3 r 3 -1 r 7 -1 r 5 h 6 1 r 4 1 r 5 3 r s 27 320 1247 e 27 320 1248 e n 333 1248 m -3 2 r -1 3 r 3 v 1 3 r 3 3 r 4 1 r 2 h 4 -1 r 3 -3 r 1 -3 r -3 v -1 -3 r -3 -2 r s n 329 1256 m 1 2 r 3 3 r 4 1 r 2 h 4 -1 r 3 -3 r 1 -2 r s 5 320 1243 d 18 8 329 1272 i 15 6 329 1274 i n 329 1288 m 18 -8 r 5 -3 r 3 -2 r 1 -3 r -1 v -1 -1 r -2 1 r 2 1 r s 7 329 1270 d 7 329 1283 d n 320 1298 m 22 h 4 1 r 1 3 r 2 v -1 3 r -3 1 r s n 320 1299 m 22 h 4 2 r 1 1 r s 10 329 1294 d n 337 1316 m 15 v -3 h -2 -1 r -2 -1 r -1 -3 r -4 v 1 -3 r 3 -3 r 4 -1 r 2 h 4 1 r 3 3 r 1 3 r 3 v -1 4 r -3 2 r s n 337 1330 m -4 h -3 -1 r s n 329 1322 m 1 -2 r 3 -3 r 4 -1 r 2 h 4 1 r 3 3 r 1 2 r s n 332 1352 m -3 1 r 5 h -2 -1 r -2 -1 r -1 -3 r -5 v 1 -3 r 2 -1 r 2 h 1 1 r 2 3 r 2 6 r 2 3 r 1 1 r s n 333 1339 m 1 1 r 1 3 r 3 6 r 1 3 r 2 1 r 3 h 2 -1 r 1 -3 r -5 v -1 -2 r -2 -2 r -2 -1 r 5 h -3 1 r s 356 1360 315 1383 b n 332 1402 m -3 1 r 5 h -2 -1 r -2 -1 r -1 -3 r -5 v 1 -3 r 2 -1 r 2 h 1 1 r 2 3 r 2 6 r 2 3 r 1 1 r s n 333 1389 m 1 1 r 1 3 r 3 6 r 1 3 r 2 1 r 3 h 2 -1 r 1 -3 r -5 v -1 -2 r -2 -2 r -2 -1 r 5 h -3 1 r s n 337 1412 m 16 v -3 h -2 -2 r -2 -1 r -1 -2 r -4 v 1 -4 r 3 -3 r 4 -1 r 2 h 4 1 r 3 3 r 1 4 r 2 v -1 4 r -3 3 r s n 337 1426 m -4 h -3 -1 r s n 329 1419 m 1 -3 r 3 -2 r 4 -2 r 2 h 4 2 r 3 2 r 1 3 r s n 333 1451 m 1 -1 r 1 1 r -1 1 r -1 h -3 -2 r -1 -3 r -4 v 1 -4 r 3 -2 r 4 -2 r 2 h 4 2 r 3 2 r 1 4 r 3 v -1 4 r -3 2 r s n 329 1443 m 1 -2 r 3 -3 r 4 -1 r 2 h 4 1 r 3 3 r 1 2 r s n 315 1460 m 2 2 r 4 3 r 5 3 r 7 1 r 5 h 6 -1 r 6 -3 r 3 -3 r 3 -2 r s n 317 1462 m 6 3 r 3 1 r 7 2 r 5 h 6 -2 r 4 -1 r 5 -3 r s 579 675 569 569 b 586 757 579 675 b 22 -2 575 770 i 22 -2 575 771 i n 585 770 m -1 3 r -1 2 r 2 v 1 3 r 3 2 r 3 h 2 h 3 -1 r 2 -2 r -3 v -3 v -1 -1 r -2 -2 r s n 583 777 m 1 2 r 2 2 r 4 h 2 h 3 -1 r 1 -2 r 1 -2 r s 4 575 767 d 22 -2 577 792 i 22 -2 577 793 i 4 577 789 d 1 7 598 787 i n 586 807 m 1 -4 r 1 -2 r 3 -1 r 2 h 4 h 2 2 r 1 3 r 2 v 3 v -2 3 r -3 1 r -2 h -3 -1 r -3 -1 r -1 -3 r c s n 586 807 m 1 -3 r 1 -2 r 3 -1 r 3 h 3 h 2 2 r 1 2 r s n 600 807 m 2 v -2 3 r -3 1 r -2 h -4 -1 r -2 -1 r -1 -2 r s n 591 833 m 1 -1 r 1 1 r 1 v -1 h -3 -2 r -1 -2 r -3 v 1 -3 r 1 -2 r 3 -2 r 2 h 4 1 r 2 2 r 1 3 r 2 v 3 v -2 2 r s n 588 827 m 1 -2 r 1 -2 r 3 -2 r 2 h 4 1 r 2 2 r 1 2 r s 22 -2 582 843 i 22 -2 582 844 i 10 -12 590 854 i 9 5 596 848 i 9 5 596 847 i 4 582 840 d 1 6 590 851 i 1 7 603 838 i 1 6 604 849 i 4 597 865 d 9 -125 597 869 i 9 35 606 744 i 7 23 615 779 i 21 -7 613 817 i 20 -7 614 818 i n 623 814 m -1 3 r 2 v 2 v 2 3 r 3 1 r 3 h 2 h 3 -2 r 1 -3 r -3 v -1 -2 r -1 -2 r -3 -1 r s n 622 821 m 2 2 r 3 1 r 3 h 2 h 2 -2 r 2 -3 r -2 v s 2 4 612 814 i 21 -7 620 837 i 21 -7 620 838 i 1 4 619 834 i 2 6 640 828 i n 632 850 m -4 v 1 -2 r 3 -2 r 2 -1 r 3 h 2 2 r 2 2 r 1 2 r 3 v -1 3 r -3 2 r -2 1 r -3 h -3 -2 r -2 -2 r c s n 632 850 m -3 v 1 -2 r 3 -2 r 2 -1 r 3 h 3 2 r 1 1 r s n 646 847 m 2 v -2 3 r -2 2 r -2 1 r -3 h -3 -2 r -2 -1 r s n 643 874 m 1 -1 r 1 h -1 2 r -1 h -2 -1 r -2 -2 r -1 -3 r -3 v 1 -3 r 3 -2 r 2 h 3 h 3 1 r 2 3 r 2 v 3 v -1 2 r s n 638 869 m -2 v 2 -3 r 2 -2 r 2 h 4 h 2 1 r 2 2 r s 21 -7 636 886 i 21 -7 636 887 i 7 -13 646 895 i 9 3 651 888 i 10 3 650 887 i 1 4 635 883 i 2 6 645 892 i 2 7 656 876 i 2 6 659 887 i 6 20 655 903 i 10 25 661 923 i 9 16 671 948 i 9 31 680 964 i 9 17 689 995 i 10 18 698 1012 i 9 13 708 1030 i 9 1 717 1043 i 9 25 726 1044 i 11 -18 737 1084 i 12 -19 737 1085 i n 743 1076 m 3 v 2 2 r 1 1 r 4 h 2 h 3 -2 r 1 -2 r 1 -3 r -1 -3 r -2 -3 r -2 -1 r -2 h -3 1 r s n 746 1082 m 3 h 3 -1 r 2 -2 r 1 -2 r 1 -3 r -1 -3 r -1 -2 r s 3 2 734 1083 i 11 -18 755 1095 i 11 -19 756 1096 i 4 2 752 1094 i 6 4 764 1075 i n 772 1097 m -2 -2 r -1 -3 r 1 -3 r 1 -2 r 2 -2 r 3 -1 r 3 1 r 2 1 r 2 2 r 1 3 r -1 3 r -1 2 r -2 2 r -3 1 r -3 -1 r c s n 772 1097 m -1 -2 r -1 -3 r 1 -3 r 1 -2 r 2 -2 r 3 h 2 h s n 781 1086 m 1 2 r 1 3 r -1 3 r -1 2 r -2 2 r -3 h -2 h s n 796 1108 m -1 v 2 -1 r 2 v -1 1 r -2 h -3 h -2 -1 r -3 -3 r -3 v -3 v 2 -2 r 2 -2 r 3 h 3 h 2 1 r 2 3 r 1 3 r s n 790 1108 m -2 -2 r -3 v -3 v 1 -2 r 3 -2 r 3 -1 r 2 h s 11 -19 799 1122 i 11 -18 800 1122 i -3 -14 812 1121 i 10 -4 811 1114 i 10 -4 810 1113 i 4 2 796 1120 i 5 3 810 1120 i 6 3 808 1102 i 6 3 817 1108 i 4 3 823 1122 i 9 4 827 1125 i 10 10 836 1129 i 9 -1 846 1139 i 9 1 855 1138 i 37 12 864 1139 i 37 21 901 1151 i 37 6 938 1172 i 20 -11 975 1178 i -7 -21 1007 1177 i -7 -20 1008 1176 i n 1005 1167 m 2 1 r 2 h 2 h 3 -2 r 1 -3 r 1 -3 r -1 -2 r -2 -3 r -2 -1 r -4 h -2 1 r -1 1 r -2 3 r s n 1011 1168 m 2 -2 r 1 -2 r 1 -4 r -1 -2 r -2 -2 r -2 -2 r -3 h s 4 -2 1004 1178 i -6 -20 1027 1170 i -6 -21 1028 1170 i 4 -1 1024 1171 i 7 -2 1018 1151 i n 1040 1159 m -3 h -3 -1 r -2 -3 r -2 v -3 v 1 -3 r 3 -2 r 2 h 3 h 3 1 r 1 3 r 1 2 r 3 v -1 3 r -3 2 r c s n 1040 1159 m -2 h -3 -2 r -2 -2 r -2 v -4 v 1 -2 r 2 -2 r s n 1038 1145 m 2 h 3 1 r 1 3 r 1 2 r 3 v -1 3 r -2 2 r s n 1065 1148 m -2 h 1 -2 r 1 1 r 1 1 r -2 3 r -1 1 r -3 1 r -4 h -2 -1 r -2 -3 r -1 -2 r -3 v 2 -3 r 2 -2 r 2 h 4 h 2 1 r s n 1060 1153 m -3 h -2 -2 r -2 -2 r -1 -2 r -3 v 2 -3 r 1 -2 r s -7 -21 1077 1156 i -6 -20 1077 1155 i -13 -6 1085 1145 i 3 -10 1079 1141 i 3 -9 1078 1141 i 3 -1 1074 1156 i 6 -1 1082 1146 i 7 -2 1067 1136 i 6 -2 1078 1133 i 30 30 1092 1138 i 37 -13 1122 1168 i 33 103 1159 1155 i 9 -20 1194 1272 i 9 -19 1195 1272 i n 1199 1263 m 2 3 r 1 1 r 2 1 r 3 h 3 -1 r 2 -2 r 1 -2 r -3 v -1 -3 r -2 -2 r -2 -1 r -3 h -2 1 r s n 1204 1268 m 2 h 3 -1 r 2 -2 r 1 -2 r -4 v -1 -2 r -1 -2 r s 1195 1272 1192 1271 b 9 -19 1214 1280 i 9 -20 1215 1281 i 4 2 1211 1279 i 1227 1262 1220 1259 b n 1231 1280 m -2 -2 r -1 -3 r -3 v 1 -2 r 2 -2 r 3 -2 r 3 1 r 2 1 r 2 2 r 1 2 r 4 v -1 2 r -2 2 r -3 1 r -3 h c s n 1231 1280 m -1 -2 r -1 -3 r -3 v 1 -2 r 2 -2 r 3 -1 r 2 h s n 1239 1268 m 1 1 r 1 3 r 3 v -1 2 r -2 3 r -3 1 r -2 h s n 1257 1288 m -2 v 1 h 1 v 1 v -3 1 r -2 h -3 -1 r -2 -2 r -1 -3 r -3 v 1 -2 r 2 -2 r 3 -1 r 3 h 2 1 r 2 2 r 1 2 r s n 1250 1288 m -1 -2 r -1 -2 r -4 v 1 -2 r 2 -2 r 3 -1 r 2 h s 9 -20 1261 1301 i 9 -20 1262 1301 i -6 -14 1275 1299 i 9 -5 1272 1291 i 9 -5 1271 1290 i 3 1 1259 1300 i 6 3 1272 1297 i 7 3 1267 1280 i 1283 1287 1278 1284 b 22 14 1284 1297 i 37 -19 1306 1311 i 37 -16 1343 1292 i 37 6 1380 1276 i 37 9 1417 1282 i 5 1454 1291 e -2 -21 1470 1302 i -2 -22 1471 1302 i n 1470 1292 m 2 1 r 2 1 r 2 h 3 -1 r 2 -3 r 1 -3 r -2 v -2 -3 r -2 -2 r -3 h -2 h -2 1 r -2 2 r s n 1476 1294 m 2 -1 r 2 -2 r 1 -4 r -2 v -2 -3 r -2 -1 r -2 -1 r s 4 1467 1302 e -2 -22 1491 1300 i -2 -22 1492 1300 i 4 1488 1300 e 1493 1278 1486 1279 b n 1506 1291 m -3 -1 r -2 -2 r -2 -3 r -2 v 1 -3 r 2 -2 r 2 -1 r 3 -1 r 3 1 r 2 2 r 1 3 r 1 2 r -1 3 r -2 2 r -3 2 r c s n 1506 1291 m -2 -1 r -2 -2 r -2 -3 r -2 v 1 -3 r 2 -2 r 1 -1 r s n 1507 1276 m 2 1 r 2 2 r 1 3 r 2 v 3 v -2 2 r -2 2 r s n 1532 1285 m -1 -1 r 1 -1 r 1 1 r 1 v -2 2 r -2 1 r -3 1 r -3 -1 r -2 -2 r -1 -3 r -1 -2 r 1 -3 r 2 -2 r 3 -2 r 2 h 3 1 r 2 2 r s n 1526 1289 m -2 -1 r -2 -2 r -1 -3 r -1 -2 r 1 -3 r 2 -2 r 2 -2 r s -3 -21 1543 1294 i -3 -21 1544 1294 i -11 -9 1553 1286 i 6 -9 1547 1280 i 5 -8 1546 1280 i 5 -1 1539 1295 i 6 1550 1286 e 7 -1 1537 1273 i 1555 1271 1548 1272 b 2 1562 1280 e 37 5 1564 1280 i 37 -21 1601 1285 i 37 -36 1638 1264 i 36 -36 1675 1228 i 3 3 1711 1192 i 34 44 1714 1195 i 10 89 569 569 i 8 83 579 658 i 1 -14 520 742 i 1 -14 521 742 i n 522 739 m 2 3 r 3 1 r 2 h 3 -1 r 1 -2 r 534 729 l s n 529 743 m 2 -1 r 1 -2 r 533 729 l s 4 517 742 e 7 518 728 e 537 729 529 728 b n 547 744 m -3 -1 r -2 -2 r -1 -4 r -2 v 2 -3 r 2 -2 r 3 h 2 h 3 1 r 2 2 r 1 3 r 2 v -2 3 r -2 2 r -3 1 r c s n 547 744 m -2 -1 r -2 -2 r -1 -4 r -2 v 2 -3 r 2 -2 r 2 h s n 550 730 m 2 1 r 2 2 r 1 3 r 2 v -2 3 r -2 2 r -2 1 r s 1 -14 563 745 i -14 565 745 d n 565 742 m 2 2 r 3 1 r 2 h 3 h 1 -2 r 577 731 l s n 572 745 m 2 -1 r 1 -2 r 576 731 l s 5 560 745 e 7 1 561 730 i 7 573 731 e 1 -21 586 753 i 1 -21 587 753 i n 587 743 m 2 2 r 2 2 r 2 h 3 -1 r 3 -2 r 1 -3 r -2 v -1 -3 r -2 -3 r -3 -1 r -2 h -2 1 r -2 2 r s n 593 747 m 2 -1 r 3 -2 r 1 -3 r -2 v -1 -3 r -2 -3 r -2 -1 r s 4 583 753 e 1 -22 607 755 i 1 -22 608 755 i 4 1 604 754 i 8 605 733 e n 623 748 m -3 -1 r -2 -2 r -1 -3 r -2 v 2 -3 r 2 -2 r 3 -1 r 2 h 3 1 r 2 3 r 1 3 r 2 v -2 3 r -2 2 r -3 1 r c s n 623 748 m -2 -1 r -2 -2 r -1 -3 r -2 v 2 -3 r 2 -2 r 2 -1 r s n 626 734 m 2 1 r 2 2 r 1 4 r 2 v -2 3 r -2 2 r -2 1 r s n 650 747 m -1 -1 r 1 -1 r 1 1 r 1 v -2 2 r -2 1 r -3 h -3 -2 r -2 -2 r -1 -3 r -2 v 1 -3 r 2 -2 r 3 -1 r 2 h 3 2 r 2 2 r s n 644 750 m -2 -2 r -2 -2 r -1 -3 r -2 v 1 -3 r 2 -2 r 2 -1 r s 1 -22 659 758 i 1 -22 660 758 i -9 -11 670 751 i 7 -8 665 745 i 7 -8 664 745 i 5 1 655 757 i 6 667 751 e 7 657 736 e 6 668 737 e 2 -25 604 742 i 9 35 606 717 i 10 34 615 752 i 9 44 625 786 i 5 30 634 830 i 14 -4 637 872 i 14 -4 637 873 i n 640 872 m -2 2 r 4 v 1 2 r 1 2 r 2 1 r 11 -3 r s n 639 880 m 1 1 r 2 1 r 11 -3 r s 1 4 636 869 i 2 7 650 865 i 654 883 652 876 b n 643 898 m -4 v 2 -2 r 2 -2 r 2 h 4 h 2 1 r 2 3 r 2 v 3 v -1 3 r -3 2 r -2 h -3 h -3 -2 r -2 -2 r c s n 643 898 m -3 v 2 -2 r 3 -2 r 2 h 3 h 2 1 r 2 2 r s n 657 896 m 2 v -2 3 r -2 2 r -2 h -4 h -2 -2 r -2 -1 r s 14 -4 647 914 i 14 -4 647 915 i n 650 914 m -2 2 r 4 v 1 2 r 1 2 r 3 1 r 11 -3 r s n 649 922 m 1 1 r 2 1 r 11 -3 r s 1 4 646 911 i 2 7 660 907 i 664 925 663 918 b 21 -5 645 937 i 21 -5 645 938 i n 655 936 m -1 2 r -1 3 r 1 2 r 1 2 r 3 2 r 3 h 2 h 3 -2 r 1 -3 r 1 -3 r -1 -2 r -1 -2 r -3 -1 r s n 654 943 m 1 1 r 3 2 r 3 h 2 h 3 -2 r 1 -3 r 1 -2 r s 645 938 644 934 b 21 -5 650 958 i 21 -5 650 959 i 1 4 649 955 i 672 957 670 950 b n 661 972 m -4 v 1 -2 r 3 -2 r 2 h 3 h 3 1 r 2 3 r 2 v 3 v -2 3 r -2 2 r -2 h -4 h -2 -2 r -2 -2 r c s n 661 972 m -3 v 2 -2 r 2 -2 r 2 h 4 h 2 1 r 2 2 r s n 675 970 m 2 v -2 3 r -3 2 r -2 h -3 h -2 -2 r -2 -1 r s n 670 997 m 1 -1 r 1 h -1 2 r -1 h -2 -2 r -2 -1 r -1 -3 r 1 -4 r 1 -2 r 3 -2 r 2 h 3 h 3 1 r 1 3 r 1 2 r 3 v -2 3 r s n 665 992 m 1 -3 r 2 -2 r 2 -2 r 2 h 4 h 2 1 r 1 2 r s 21 -5 662 1008 i 21 -5 662 1009 i 7 -13 672 1018 i 9 4 677 1011 i 10 4 676 1010 i 1 4 661 1005 i 1 6 671 1015 i 2 7 682 1000 i 686 1017 685 1011 b 1 4 679 1027 i 9 24 680 1031 i 9 689 1055 e 10 6 698 1055 i 9 15 708 1061 i 9 11 717 1076 i 9 9 726 1087 i 9 7 735 1096 i 10 -24 744 1103 i 9 -18 754 1079 i 9 2 763 1061 i 9 2 772 1063 i 1 781 1065 d 13 -5 758 1023 i 13 -5 758 1024 i n 761 1023 m -1 3 r 3 v 1 2 r 2 2 r 2 h 11 -4 r s n 761 1031 m 2 1 r 2 h 10 -4 r s 2 4 756 1020 i 3 6 770 1015 i 777 1032 774 1025 b n 768 1048 m -3 v 1 -3 r 2 -2 r 2 -1 r 4 h 2 1 r 2 2 r 1 2 r 3 v -1 3 r -2 2 r -2 1 r -3 h -3 -1 r -2 -2 r c s n 768 1048 m -2 v 1 -3 r 3 -2 r 2 -1 r 3 h 3 1 r 1 1 r s n 782 1044 m 2 v -1 3 r -3 2 r -1 1 r -4 h -2 -1 r -2 -1 r s 13 -6 775 1063 i 13 -6 775 1064 i n 778 1063 m -1 2 r 4 v 1 2 r 2 2 r 2 h 11 -4 r s n 778 1071 m 1 1 r 3 h 10 -4 r s 2 4 773 1060 i 2 6 787 1055 i 794 1072 791 1065 b 20 -9 777 1087 i 20 -9 777 1088 i n 787 1084 m -1 2 r -1 3 r 1 1 r 2 3 r 3 1 r 3 h 2 -1 r 3 -2 r 1 -3 r -1 -3 r -2 v -2 -2 r -3 -1 r s n 786 1090 m 2 2 r 3 1 r 3 h 2 -1 r 2 -2 r 1 -3 r -2 v s 777 1088 776 1084 b 20 -8 785 1106 i 19 -8 786 1107 i 2 3 784 1104 i 807 1102 804 1095 b n 798 1118 m -3 v 1 -3 r 2 -2 r 2 -1 r 3 h 3 1 r 2 2 r 1 2 r 3 v -1 3 r -2 2 r -2 1 r -4 h -2 -1 r -2 -2 r c s n 798 1118 m -2 v 1 -3 r 2 -2 r 2 -1 r 4 h 2 1 r 2 1 r s n 812 1114 m 2 v -1 3 r -3 2 r -2 1 r -3 h -3 -1 r -1 -1 r s n 811 1141 m 1 -1 r 1 h -1 2 r -1 h -2 -1 r -2 -1 r -1 -3 r -3 v 1 -3 r 2 -2 r 2 -1 r 3 h 3 1 r 2 2 r 1 2 r 3 v -1 3 r s n 806 1137 m -2 v 1 -3 r 2 -2 r 2 -1 r 4 h 2 1 r 2 1 r s 20 -9 805 1154 i 19 -9 806 1155 i 6 -13 816 1161 i 10 3 820 1154 i 9 3 820 1153 i 2 4 804 1151 i 2 5 815 1159 i 3 6 824 1143 i 831 1159 828 1153 b 7 70 802 1115 i 9 -5 809 1185 i 9 21 818 1180 i 9 7 827 1201 i 10 -19 836 1208 i 9 20 846 1189 i 9 5 855 1209 i 2 -4 864 1214 i -7 -12 865 1216 i -7 -12 866 1216 i n 864 1213 m 3 1 r 3 -1 r 2 -1 r 2 -2 r -3 v -6 -9 r s n 872 1212 m 1 -2 r -2 v -6 -10 r s 3 -2 863 1218 i 6 -4 855 1206 i 871 1196 865 1200 b n 888 1202 m -3 1 r -3 -1 r -3 -2 r -1 -1 r -4 v -2 v 2 -3 r 2 -1 r 3 -1 r 3 1 r 3 2 r 1 2 r 3 v 3 v -2 2 r c s n 888 1202 m -2 1 r -3 -1 r -3 -2 r -1 -2 r -1 -3 r 1 -3 r 1 -2 r s n 882 1189 m 2 h 3 h 3 3 r 1 1 r 1 3 r -1 3 r -1 2 r s -8 -13 902 1194 i -8 -12 903 1193 i n 901 1191 m 3 h 3 -1 r 2 -1 r 2 -2 r -2 v -6 -10 r s n 909 1189 m 1 -2 r -2 v -6 -9 r s 4 -2 899 1195 i 6 -4 892 1183 i 907 1173 901 1177 b -11 -18 925 1188 i -12 -18 926 1187 i n 920 1179 m 3 h 3 h 1 -1 r 2 -2 r 1 -3 r -1 -3 r -1 -2 r -2 -2 r -3 -1 r -3 1 r -2 1 r -1 2 r -1 3 r s n 927 1178 m 2 -2 r -3 v -1 -3 r -1 -2 r -2 -2 r -3 h -2 h s 926 1187 922 1190 b -11 -19 943 1177 i -11 -18 944 1176 i 3 -2 941 1178 i 935 1156 929 1160 b n 953 1162 m -4 1 r -2 -1 r -3 -2 r -1 -1 r -1 -4 r 1 -2 r 2 -3 r 2 -1 r 3 -1 r 3 1 r 2 2 r 1 2 r 1 3 r -1 3 r -2 2 r c s n 953 1162 m -3 1 r -2 -1 r -3 -2 r -1 -2 r -1 -3 r 1 -3 r 1 -2 r s n 947 1149 m 2 h 3 1 r 2 2 r 2 1 r 3 v 3 v -2 2 r s n 974 1146 m -2 -1 r 1 -1 r 1 h 1 1 r -1 3 r -1 2 r -3 2 r -3 h -3 h -2 -2 r -1 -2 r -1 -3 r 1 -3 r 2 -3 r 1 -1 r 4 -1 r 2 1 r s n 970 1152 m -2 h -3 -1 r -2 -2 r -2 -2 r -3 v -3 v 2 -2 r s -11 -19 987 1150 i -11 -18 988 1149 i -14 -4 993 1138 i 1 -10 985 1135 i 1 -11 984 1136 i 4 -2 984 1151 i 6 -3 990 1139 i 6 -4 973 1133 i 988 1124 983 1127 b 24 6 987 1135 i 37 14 1011 1141 i 37 5 1048 1155 i 37 55 1085 1160 i 5 1 1122 1215 i 11 -9 1117 1212 i 11 -9 1118 1213 i n 1120 1211 m 3 v 1 3 r 1 1 r 3 2 r 2 h 9 -7 r s n 1122 1218 m 2 1 r 3 h 8 -7 r s 3 3 1115 1210 i 5 5 1126 1201 i 1138 1215 1133 1209 b n 1134 1233 m -1 -3 r -3 v 2 -3 r 1 -1 r 3 -1 r 3 h 3 2 r 1 1 r 2 3 r -1 3 r -2 3 r -1 1 r -3 1 r -3 h -3 -2 r c s n 1134 1233 m -1 -2 r 1 -3 r 1 -3 r 2 -1 r 3 -2 r 3 1 r 2 1 r s n 1146 1225 m 1 3 r 2 v -2 3 r -2 2 r -3 1 r -3 -1 r -2 -1 r s 11 -9 1144 1246 i 11 -9 1145 1246 i n 1147 1244 m 3 v 1 3 r 1 2 r 3 2 r 2 -1 r 9 -7 r s n 1149 1252 m 2 1 r 3 -1 r 9 -7 r s 3 3 1142 1243 i 5 6 1153 1234 i 1165 1249 1161 1243 b 17 -14 1153 1268 i 17 -13 1153 1268 i n 1161 1262 m 3 v 1 2 r 1 2 r 3 2 r 3 h 3 -1 r 1 -2 r 2 -2 r -3 v -1 -3 r -1 -2 r -2 -1 r -3 h s n 1163 1269 m 2 1 r 3 h 3 -1 r 2 -1 r 1 -3 r 1 -3 r -1 -2 r s 1153 1268 1151 1265 b 17 -13 1166 1284 i 17 -13 1167 1285 i 3 3 1164 1282 i 1186 1274 1181 1269 b n 1182 1292 m -1 -3 r -3 v 2 -3 r 1 -1 r 3 -1 r 3 h 3 2 r 1 2 r 1 3 r 2 v -2 3 r -1 1 r -3 2 r -3 -1 r -3 -1 r c s n 1182 1292 m -1 -2 r 1 -3 r 1 -3 r 2 -1 r 3 -1 r 3 h 2 1 r s n 1194 1285 m 1 2 r -1 3 r -1 2 r -2 2 r -3 1 r -3 h -2 -1 r s n 1201 1311 m -2 v 2 1 r -1 1 r 1 v -3 -1 r -2 -1 r -2 -2 r -2 -3 r 1 -3 r 2 -3 r 1 -1 r 3 -1 r 3 h 3 2 r 1 1 r 1 4 r 2 v s n 1195 1308 m -1 -2 r -3 v 2 -3 r 2 -1 r 3 -1 r 3 h 2 1 r s 17 -13 1199 1324 i 17 -13 1199 1325 i 2 -15 1211 1329 i 10 -1 1213 1321 i 11 1212 1320 e 2 3 1197 1322 i 4 5 1209 1326 i 4 6 1214 1308 i 1225 1322 1221 1317 b 23 -25 1210 1319 i 36 39 1233 1294 i 37 4 1269 1333 i 37 -12 1306 1337 i 9 2 1343 1325 i 5 -13 1353 1333 i 5 -14 1354 1334 i n 1355 1331 m 1 3 r 2 2 r 2 h 4 1 r 1 -2 r 4 -10 r s n 1360 1336 m 3 h 1 -1 r 5 -11 r s 4 2 1350 1332 i 7 2 1355 1319 i 1372 1326 1366 1323 b n 1378 1343 m -3 -2 r -1 -3 r -3 v 1 -2 r 2 -2 r 3 -1 r 3 h 2 1 r 2 2 r 1 2 r 4 v -1 2 r -2 2 r -2 1 r -4 h c s n 1378 1343 m -2 -1 r -1 -3 r -3 v 1 -2 r 2 -3 r 3 -1 r 2 h s n 1385 1331 m 1 1 r 1 3 r 3 v 2 v -3 3 r -2 1 r -3 h s 5 -13 1393 1349 i 5 -14 1394 1350 i n 1395 1347 m 1 2 r 3 3 r 1 h 4 1 r 1 -2 r 5 -11 r s n 1400 1352 m 3 h 1 -1 r 5 -11 r s 4 2 1390 1348 i 7 2 1395 1335 i 1412 1342 1406 1339 b 8 -20 1411 1364 i 8 -20 1412 1365 i n 1416 1355 m 1 3 r 2 2 r 2 h 3 1 r 2 -2 r 3 -2 r -2 v -3 v -1 -3 r -2 -2 r -2 -1 r -2 h -3 2 r s n 1421 1360 m 2 h 2 -1 r 3 -3 r -1 v 1 -4 r -2 -2 r -1 -2 r s 1412 1365 1408 1363 b 8 -20 1431 1372 i 8 -20 1432 1373 i 4 2 1428 1371 i 1443 1354 1436 1351 b n 1448 1371 m -2 -2 r -1 -2 r -4 v 1 -2 r 2 -2 r 2 -1 r 4 h 1 1 r 3 2 r 1 2 r 4 v -1 2 r -2 2 r -3 1 r -3 h c s n 1448 1371 m -1 -1 r -1 -3 r -3 v -2 v 3 -3 r 2 -1 r 3 h s n 1455 1359 m 2 1 r 1 3 r 3 v -1 2 r -2 3 r -3 1 r -2 h s n 1474 1378 m -1 v 1 -1 r 1 2 r -1 1 r -3 1 r -2 h -3 -1 r -2 -2 r -1 -3 r -3 v 1 -2 r 2 -2 r 2 -2 r 4 1 r 2 h 2 2 r 1 3 r s n 1467 1379 m -1 -2 r -1 -2 r -4 v 1 -2 r 2 -2 r 2 -1 r 3 h s 8 -20 1479 1391 i 8 -20 1480 1392 i -6 -14 1492 1389 i 8 -5 1490 1381 i 8 -6 1489 1381 i 4 2 1476 1390 i 6 2 1489 1388 i 7 3 1484 1370 i 1500 1377 1495 1374 b 30 16 1497 1385 i 37 -10 1527 1401 i 37 16 1564 1391 i 37 -3 1601 1407 i 23 4 1638 1404 i 87 5 1661 1408 i 10 54 569 569 i 9 50 579 623 i 9 49 588 673 i 4 23 597 722 i 21 -4 591 756 i 21 -4 591 757 i n 591 753 m 2 13 r 2 2 r 1 1 r 2 1 r 2 -1 r 2 -1 r 1 -1 r -3 v 601 755 l s n 593 766 m 1 1 r 2 1 r 2 1 r 2 -1 r 2 -1 r -1 v 1 -2 r s 1 7 612 749 i n 602 761 m 2 1 r 1 1 r 8 2 r 1 1 r 1 v -1 1 r s n 604 762 m 2 1 r 7 1 r 2 h 2 v -2 2 r -1 h s 22 -4 597 789 i 21 -4 598 790 i n 608 787 m -3 -2 r -1 -2 r -1 -2 r 1 -3 r 1 -2 r 3 -2 r 2 h 3 h 3 2 r 1 2 r 1 3 r -1 2 r -1 2 r s n 603 781 m 1 -2 r 2 -2 r 2 -2 r 2 h 4 h 2 2 r 1 1 r s 1 4 597 786 i 4 619 785 d 15 4 607 797 i 13 3 607 798 i n 609 810 m 622 801 l 3 -3 r 2 -3 r 1 -2 r -1 v -2 -1 r 2 v 1 h s 1 6 606 795 i 609 812 608 805 b 22 -4 603 819 i 21 -4 604 820 i n 603 816 m 2 12 r 2 3 r 1 1 r 2 h 2 h 2 -1 r 1 -2 r -3 v -1 -8 r s n 605 828 m 2 2 r 1 1 r 2 h 2 h 2 -2 r 1 -1 r -2 v s 1 7 624 812 i n 615 823 m 1 2 r 1 1 r 8 1 r 1 1 r 1 v 1 v s n 616 825 m 2 h 8 1 r 1 1 r 2 v -1 1 r -1 1 r s n 620 849 m 1 -1 r 1 1 r -1 1 r -1 h -2 -1 r -2 -2 r -3 v -3 v 2 -3 r 3 -1 r 2 -1 r 3 1 r 2 1 r 2 3 r 2 v 3 v -2 3 r s n 616 844 m -2 v 2 -3 r 3 -1 r 2 -1 r 3 1 r 2 1 r 2 2 r s 16 3 618 857 i 13 3 619 858 i 13 -9 621 869 i 1 6 618 855 i 621 871 620 865 b 14 -3 622 877 i 14 -3 623 878 i n 629 877 m -3 2 r -2 2 r -1 2 r 1 3 r 1 1 r 1 h 1 -1 r -1 -1 r -1 1 r s 1 4 622 874 i 637 878 636 871 b 3 22 631 893 i 9 34 634 915 i 9 40 643 949 i 9 23 652 989 i 10 5 661 1012 i 9 -2 671 1017 i 9 21 680 1015 i 5 10 689 1036 i 18 -12 681 1050 i 18 -11 682 1050 i n 680 1047 m 6 10 r 3 2 r 1 1 r 3 -1 r 1 -1 r 2 -2 r -1 v -1 -3 r -4 -7 r s n 686 1057 m 2 2 r 2 h 2 h 2 -2 r 1 -2 r -1 v -2 v s 4 6 698 1035 i n 693 1049 m 2 1 r 2 1 r 8 -2 r 1 h 1 v 2 v s n 695 1050 m 3 h 7 -2 r 1 h 1 2 r -1 2 r -1 h s 18 -12 700 1077 i 18 -12 700 1078 i n 708 1071 m -3 h -2 -1 r -1 -2 r -1 -3 r 1 -3 r 2 -3 r 2 -1 r 3 -1 r 3 1 r 2 2 r 1 2 r 1 2 r -1 3 r s n 702 1068 m -2 v -3 v 2 -3 r 2 -1 r 3 -1 r 3 1 r 2 1 r s 2 3 698 1075 i 720 1069 718 1065 b 16 -3 711 1082 i 13 -3 712 1083 i n 718 1092 m 727 1079 l 2 -4 r -3 v -2 v -1 v -2 h 1 v 1 h s 3 5 710 1080 i 719 1094 716 1089 b 18 -12 716 1103 i 18 -12 717 1104 i n 715 1100 m 7 11 r 2 2 r 2 h 2 h 2 -2 r 1 -1 r -2 v -1 -3 r -4 -7 r s n 722 1111 m 2 1 r 1 h 2 h 2 -1 r 1 -2 r -2 v -2 v s 4 6 733 1088 i n 728 1102 m 2 2 r 2 h 8 -2 r 1 1 r 1 1 r -1 1 r s n 730 1104 m 3 -1 r 7 -2 r 1 h 1 2 r -1 2 r -1 1 r s n 743 1125 m -2 v 2 1 r 1 v -1 1 r -3 -1 r -2 -1 r -2 -3 r -1 -3 r 1 -3 r 2 -2 r 2 -1 r 3 -1 r 3 h 2 2 r 1 2 r 1 3 r 3 v s n 737 1121 m -2 v -3 v 3 -2 r 1 -2 r 3 h 3 h 2 1 r s 15 -2 745 1132 i 13 -2 745 1133 i 9 -13 751 1143 i 4 5 743 1131 i 752 1144 749 1139 b 12 -8 756 1150 i 12 -7 756 1150 i n 762 1147 m -2 3 r -1 2 r 3 v 2 2 r 1 1 r 1 -1 r 1 -1 r -2 -1 r 2 v s 2 3 754 1147 i 770 1145 766 1139 b 1 1 762 1149 i 9 28 763 1150 i 9 26 772 1178 i 9 -3 781 1204 i 10 12 790 1201 i 9 9 800 1213 i 9 7 809 1222 i 9 10 818 1229 i 9 9 827 1239 i 10 5 836 1248 i 9 4 846 1253 i 9 20 855 1257 i 2 864 1277 e 9 -19 869 1292 i 9 -20 869 1293 i n 866 1291 m 11 5 r 3 h 2 h 1 -2 r 1 -1 r -3 v -1 v -3 -2 r -7 -4 r s n 877 1296 m 2 h 2 -1 r 1 -1 r 1 -2 r -2 v -1 v -2 -2 r s 6 3 875 1271 i n 878 1285 m 3 h 1 h 6 -5 r 1 -1 r 1 1 r 1 1 r s n 881 1285 m 2 -1 r 4 -6 r 2 h 2 1 r 2 v -1 1 r s 9 -20 898 1306 i 9 -19 899 1306 i n 903 1297 m -3 1 r -2 h -2 -1 r -3 -2 r -1 -3 r 1 -3 r 1 -2 r 2 -3 r 3 -1 r 3 1 r 2 1 r 1 1 r 1 3 r s n 896 1297 m -2 -2 r -1 -3 r 1 -3 r 1 -2 r 2 -2 r 3 -1 r 2 h s 3 1 896 1305 i 911 1288 907 1286 b 11 -11 911 1304 i 10 -9 912 1304 i n 922 1309 m -16 v -4 v -1 -3 r -1 -2 r -1 h -2 h 1 1 r 1 h s 6 2 909 1303 i 924 1310 918 1307 b 9 -20 926 1319 i 9 -19 927 1319 i n 924 1317 m 11 6 r 3 h 1 -1 r 2 -1 r 1 -2 r -2 v -2 v -3 -2 r -7 -3 r s n 935 1323 m 2 h 2 -1 r 1 -1 r 1 -2 r -3 v -1 v -2 -2 r s 6 3 933 1298 i n 936 1312 m 3 h 1 -1 r 6 -5 r 1 h 1 h 1 1 r s n 939 1312 m 1 -2 r 5 -5 r 2 -1 r 2 1 r 2 v -1 1 r s n 961 1323 m -1 -1 r 2 -1 r 2 v 1 v -3 1 r -2 h -3 -2 r -3 -2 r -1 -3 r 1 -3 r 1 -2 r 2 -2 r 3 -1 r 3 h 2 1 r 2 2 r 1 3 r s n 954 1323 m -2 -1 r -1 -3 r 1 -3 r 1 -2 r 2 -3 r 2 -1 r 3 h s 12 -11 966 1329 i 10 -9 967 1329 i 1 -16 977 1334 i 6 3 964 1328 i 979 1335 973 1332 b 6 -13 985 1337 i 6 -13 986 1338 i n 988 1332 m 4 v 1 2 r 1 2 r 3 1 r 1 h 1 -1 r -1 -1 r -1 h 1 v s 4 2 982 1336 i 994 1326 988 1323 b 10 8 1001 1339 i 37 -3 1011 1347 i 37 14 1048 1344 i 37 23 1085 1358 i 37 1122 1381 e 7 6 1159 1381 i 20 -6 1156 1396 i 20 -6 1156 1397 i n 1155 1393 m 3 12 r 2 3 r 2 h 2 1 r 2 -1 r 1 -2 r 1 -1 r -3 v -2 -8 r s n 1158 1405 m 2 2 r 1 h 2 1 r 2 -1 r 2 -2 r 1 -1 r -2 v s 2 7 1175 1387 i n 1167 1399 m 2 2 r 1 h 8 1 r 1 1 r 1 v 1 v s n 1169 1401 m 2 h 8 h 1 h 2 v -1 2 r -1 h s 21 -6 1165 1427 i 20 -6 1166 1428 i n 1175 1424 m -2 -1 r -2 -2 r -1 -2 r -3 v 2 -2 r 2 -2 r 2 -1 r 4 h 2 1 r 2 3 r 1 2 r -1 2 r -1 3 r s n 1170 1419 m 1 -2 r 1 -2 r 3 -2 r 2 -1 r 3 h 3 1 r 1 2 r s 2 4 1164 1424 i 1187 1425 1186 1421 b 16 2 1175 1435 i 13 1 1176 1436 i n 1179 1447 m 12 -10 r 3 -3 r 2 -3 r -2 v -1 v -2 -1 r 1 v 1 1 r s 2 6 1175 1433 i 1180 1449 1178 1443 b 20 -7 1175 1457 i 20 -7 1175 1458 i n 1174 1454 m 3 12 r 2 2 r 2 1 r 2 h 2 h 1 -2 r 1 -1 r -3 v -2 -8 r s n 1177 1466 m 2 1 r 1 1 r 2 h 2 h 2 -2 r 1 -1 r -2 v s 2 6 1194 1448 i n 1186 1460 m 2 1 r 1 1 r 8 1 r 1 1 r 2 v s n 1188 1461 m 2 1 r 8 h 1 h 2 v -1 2 r -1 h s n 1194 1485 m 1 -1 r 1 1 r -1 1 r -1 h -2 -1 r -2 -2 r -1 -3 r -3 v 2 -3 r 2 -1 r 2 -1 r 4 h 2 1 r 2 3 r 1 2 r 3 v -2 3 r s n 1189 1480 m 1 -2 r 1 -3 r 3 -1 r 2 -1 r 3 h 3 1 r 1 2 r s 16 2 1193 1493 i 13 1 1194 1494 i 12 -10 1197 1505 i 2 6 1193 1491 i 1198 1507 1196 1501 b 14 -5 1199 1513 i 14 -5 1200 1514 i n 1206 1512 m -3 2 r -1 2 r -1 3 r 1 2 r 2 1 r 1 h -1 v -1 -1 r -1 1 r s 1 4 1199 1510 i 1214 1512 1212 1505 b 21 177 1208 1524 i -13 -17 1210 1731 i -13 -17 1211 1730 i n 1208 1732 m 9 -7 r 2 -3 r -1 v -3 v -1 -1 r -3 -1 r -1 h -3 1 r -7 5 r s n 1217 1725 m 1 -2 r -2 v -2 v -1 -1 r -2 -1 r -2 -1 r -2 1 r s 6 -4 1194 1715 i n 1209 1719 m 1 -2 r -2 v -2 -7 r -2 v 1 h 1 h s n 1210 1717 m -1 -3 r -3 -7 r -1 v 2 -1 r 2 1 r 1 h s -13 -17 1236 1710 i -13 -17 1237 1710 i n 1230 1702 m -1 3 r -1 2 r -1 1 r -3 2 r -3 -1 r -3 -2 r -1 -1 r -1 -3 r -3 v 2 -3 r 1 -1 r 3 -1 r 3 1 r s n 1227 1708 m -3 1 r -2 h -3 -2 r -1 -2 r -2 -3 r 1 -3 r 1 -2 r s 3 -2 1234 1712 i 1226 1691 1223 1693 b -4 -15 1240 1698 i -3 -13 1240 1698 i n 1249 1691 m 1236 1683 l -4 -2 r -3 h -3 h 1 v -1 1 r 2 1 r -2 v s 5 -4 1238 1700 i 1251 1690 1246 1693 b -13 -17 1260 1691 i -13 -17 1261 1691 i n 1258 1693 m 10 -7 r 1 -3 r 1 -1 r -1 -3 r -1 -1 r -2 -1 r -2 h -3 1 r -6 5 r s n 1268 1686 m 1 -2 r -2 v -1 -2 r -1 -2 r -2 -1 r -1 h -3 1 r s 5 -4 1245 1676 i n 1259 1680 m 1 -3 r -1 v -2 -8 r -1 v 1 -1 r 2 1 r s n 1260 1677 m -1 -2 r -2 -7 r -1 v 1 -1 r 3 1 r s n 1280 1663 m -1 h -1 v 1 h 1 1 r -1 2 r -1 3 r -2 1 r -3 1 r -3 h -3 -2 r -1 -1 r -1 -3 r -3 v 2 -3 r 2 -1 r 3 -1 r 3 h s n 1277 1669 m -2 1 r -3 -1 r -3 -1 r -1 -2 r -1 -3 r -3 v 1 -2 r s -4 -15 1288 1661 i -3 -13 1288 1661 i -13 -8 1297 1654 i 5 -3 1286 1662 i 1299 1652 1294 1656 b -9 -12 1304 1649 i -9 -11 1305 1648 i n 1301 1643 m 3 2 r 2 h 3 h 2 -2 r -2 v -2 -1 r 2 v 1 h s 4 -2 1301 1650 i 1298 1635 1293 1639 b 25 -176 1280 1660 i -8 -20 1305 1497 i -8 -20 1306 1497 i n 1303 1498 m 11 -5 r 2 -2 r 1 -1 r -2 v -1 -2 r -1 -2 r -2 h -3 h -8 3 r s n 1314 1493 m 1 -1 r 1 -2 r -2 v -1 -2 r -1 -1 r -2 -1 r -2 h s 7 -2 1294 1478 i n 1307 1485 m 2 -2 r -1 v -8 v 1 -1 r 1 -1 r 1 1 r s n 1309 1483 m -2 v -1 -7 r -2 v 2 h 2 1 r 1 v s -9 -20 1336 1484 i -9 -20 1337 1484 i n 1332 1475 m -1 2 r -2 2 r -2 1 r -3 h -3 -1 r -2 -2 r -1 -2 r -3 v 1 -3 r 3 -2 r 2 -1 r 2 h 3 1 r s n 1327 1480 m -2 h -3 -1 r -2 -3 r -1 -2 r -3 v 1 -3 r 2 -1 r s 4 -1 1333 1485 i 1331 1463 1327 1464 b -16 1342 1474 d -13 1343 1473 d n 1354 1469 m -12 -11 r -3 -3 r -3 -1 r -2 h -1 h -1 1 r 2 1 r -1 v s 5 -2 1341 1474 i 1356 1468 1350 1470 b -8 -20 1364 1472 i -8 -20 1365 1472 i n 1361 1473 m 12 -4 r 2 -3 r 1 -1 r -2 v -1 -2 r -2 -2 r -1 h -3 h -8 3 r s n 1373 1469 m 1 -2 r 1 -2 r -2 v -1 -2 r -2 -1 r -1 -1 r -2 h s 7 -3 1353 1454 i n 1366 1460 m 1 -1 r 1 -2 r -8 v -1 v 1 h 2 h s n 1367 1459 m -3 v -7 v -2 v 2 h 2 1 r 1 v s n 1390 1450 m -1 -1 r 1 -1 r 1 1 r -1 3 r -1 2 r -3 1 r -3 h -3 -1 r -2 -2 r -1 -2 r -3 v 1 -3 r 2 -2 r 2 -1 r 3 h 3 1 r s n 1386 1455 m -2 h -3 -1 r -2 -3 r -1 -1 r -4 v 1 -2 r 1 -2 r s -16 1398 1450 d -14 1399 1450 d -12 -11 1410 1445 i 6 -3 1396 1451 i 1412 1444 1406 1447 b -5 -13 1417 1442 i -5 -14 1418 1442 i n 1416 1436 m 2 2 r 3 1 r 2 1 r 3 -2 r -1 v -1 v -1 -1 r -1 2 r 1 h s 4 -1 1414 1443 i 1415 1427 1409 1430 b 36 14 1418 1436 i 36 4 1454 1450 i 37 2 1490 1454 i 37 2 1527 1456 i 29 2 1564 1458 i 3 -22 1600 1474 i 3 -21 1601 1474 i n 1597 1473 m 12 2 r 3 -1 r 2 h 1 -2 r -2 v -1 -3 r -1 v -3 -1 r -9 -1 r s n 1609 1475 m 2 -1 r 2 -1 r 1 -1 r -3 v -1 -2 r -1 v -2 -1 r s 7 1 1600 1452 i n 1608 1464 m 2 h 1 -1 r 4 -7 r 1 -1 r 1 h 1 2 r s n 1610 1464 m 1 -2 r 3 -7 r 1 -1 r 2 h 1 3 r 1 v s 3 -21 1633 1478 i 3 -21 1634 1478 i n 1634 1468 m -2 2 r -2 1 r -2 -1 r -3 -1 r -2 -2 r -1 -4 r 1 -2 r 1 -3 r 2 -1 r 3 -1 r 3 h 1 2 r 2 2 r s n 1628 1470 m -2 -1 r -2 -2 r -1 -4 r 1 -2 r 1 -2 r 2 -2 r 2 -1 r s 4 1630 1478 e 4 1636 1457 e 8 -13 1644 1472 i 7 -12 1645 1473 i n 1656 1474 m 1652 1459 l -2 -4 r -1 -3 r -2 -1 r -1 h -1 1 r 1 v 2 -1 r s 6 1 1642 1472 i 6 1652 1474 e 3 -21 1663 1482 i 3 -22 1664 1483 i n 1660 1482 m 12 2 r 4 -1 r 1 -1 r 1 -2 r -2 v -2 v -1 -1 r -3 -2 r -8 -1 r s n 1672 1484 m 3 -1 r 1 -1 r 1 -2 r -2 v -2 v -1 -1 r -2 -2 r s 7 1 1663 1461 i n 1671 1473 m 2 -1 r 1 -1 r 4 -6 r 1 -1 r 1 h 1 1 r s n 1673 1472 m 1 -2 r 3 -6 r 1 -1 r 2 h 1 2 r 1 v s n 1697 1477 m -1 -2 r 1 h 1 1 r 1 v -2 2 r -2 h -3 h -3 -2 r -2 -2 r -1 -3 r 1 -2 r 1 -3 r 2 -2 r 4 h 2 h 3 1 r 1 3 r s n 1691 1479 m -2 -1 r -2 -3 r -1 -3 r 1 -2 r 1 -3 r 2 -2 r 3 h s 8 -14 1704 1481 i 7 -12 1705 1481 i -4 -15 1716 1482 i 6 1 1702 1480 i 1718 1483 1712 1482 b 2 -14 1724 1483 i 2 -15 1725 1484 i n 1726 1477 m 1 4 r 2 2 r 1 1 r 3 1 r 2 -1 r -1 v -1 -1 r -1 1 r 1 1 r s 4 1 1721 1483 i 1730 1470 1723 1469 b 7 3 1741 1481 i p end %%EndDocument @endspecial 98 x Fa(Figure)15 b(2:)k(Comm)o(unications)c(rates)f(for)h (\\buc)o(k)o(et-brigade")f(messages)h(in)g(a)g(ring)g(of)f(four)g(pro)q (cessors)74 2287 y(on)h(an)g(In)o(tel)h(ipsc/860.)963 2790 y(3)p eop %%Trailer end userdict /end-hook known{end-hook}if %%EOF From owner-mpi-pt2pt@CS.UTK.EDU Tue Jan 5 17:49:51 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA08019; Tue, 5 Jan 93 17:49:51 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA27933; Tue, 5 Jan 93 17:49:38 -0500 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Tue, 05 Jan 1993 22:49:37 GMT Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from [132.175.13.2] by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA27924; Tue, 5 Jan 93 17:49:36 -0500 Received: from panther.cs.sandia.gov by cs.sandia.gov (4.1/SMI-4.1) id AA22393; Tue, 5 Jan 93 15:49:25 MST Received: by panther.cs.sandia.gov (Smail3.1.28.1 #1) id m0n9N5Q-0016ZKC; Tue, 5 Jan 93 15:49 MST Message-Id: Date: Tue, 5 Jan 93 15:49 MST From: srwheat@cs.sandia.gov (Stephen R. Wheat) To: mpi-pt2pt@cs.utk.edu Subject: Re: Comments on "ready receiver" part of draft implementation We would like to second the position taken by Gropp and Lusk. With our Sandia/Univ. of New Mexico OS (SUNMOS) project, we have also discovered the value of posting non-blocking receives for anticipated messages. In SUNMOS 1, we provided this facility at the kernel level. However, we also saw the benefit of having permanently posted receives. That is, once such a receive is posted, any matching message would get delivered to the specified location (overwriting previous data), with an associated integer flag being incremented upon message receipt. Many applications utilize repetitive boundary data exchanges; this strategy avoids the costs of independent receives. We have proceeded to a new kernel design that uses posted receives as the basis for all message communication. A message passing design document is currently in the works :-). We will be happy to electronically forward that document to you as it becomes available. Stephen Wheat and Barney Maccabe SNL UNM/SNL From owner-mpi-profile@CS.UTK.EDU Wed Jan 13 17:46:23 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA11465; Wed, 13 Jan 93 17:46:23 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA01528; Wed, 13 Jan 93 17:25:49 -0500 X-Resent-To: mpi-profile@CS.UTK.EDU ; Wed, 13 Jan 1993 17:25:47 EST Errors-To: owner-mpi-profile@CS.UTK.EDU Received: from THUD.CS.UTK.EDU by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA01521; Wed, 13 Jan 93 17:25:46 -0500 From: Jack Dongarra Received: by thud.cs.utk.edu (5.61++/2.7c-UTK) id AA01688; Wed, 13 Jan 93 17:25:45 -0500 Date: Wed, 13 Jan 93 17:25:45 -0500 Message-Id: <9301132225.AA01688@thud.cs.utk.edu> To: mpi-profile@cs.utk.edu Subject: this is a test testing From owner-mpi-pt2pt@CS.UTK.EDU Fri Jan 15 12:26:48 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA14370; Fri, 15 Jan 93 12:26:48 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA03545; Fri, 15 Jan 93 12:25:16 -0500 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Fri, 15 Jan 1993 12:25:14 EST Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from fslg8.fsl.noaa.gov by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA03526; Fri, 15 Jan 93 12:25:09 -0500 Received: by fslg8.fsl.noaa.gov (5.57/Ultrix3.0-C) id AA19456; Fri, 15 Jan 93 17:24:54 GMT Received: by macaw.fsl.noaa.gov (4.1/SMI-4.1) id AA14219; Fri, 15 Jan 93 10:23:53 MST Date: Fri, 15 Jan 93 10:23:53 MST From: hender@macaw.fsl.noaa.gov (Tom Henderson) Message-Id: <9301151723.AA14219@macaw.fsl.noaa.gov> To: mpi-pt2pt@cs.utk.edu Subject: "Message Capsule" Proposal (long) Hi all, Here is a proposal that we think may help reduce the number of separate send and receive routines to something much less than "512" without eliminating any of the nice features that we haven't been able to say "no" to. The proposal also allows for "simple" versions of send and receive routines for novice users. The proposal borrows ideas from Jim Cownie, Paul Pierce, and others (without their consent-- don't blame anyone but us!). We hope that this proposal will at least start a few good arguments. :) Leslie Hart hart@fsl.noaa.gov Tom Henderson hender@fsl.noaa.gov Bernardo Rodriguez bernardo@fsl.noaa.gov NOAA Forecast Systems Lab ----------------------------------------------------------------------------- A PROPOSAL FOR "MESSAGE CAPSULES" 1 Introduction to Message Capsules To send a message, a user must first create and initialize a data structure called a "message capsule". (In Fortran77 a message capsule might be an integer array, possibly containing pointers used by underlying C routines. Another way might be to use the POSIX approach mentioned by Bob Knighten.) The message capsule is an opaque object: a user should only access the contents of a message capsule using a special set of routines. A message capsule contains a complete description of the user data in the message including user data location, data type(s), and optional stride information. The capsule will also contain information describing the data in an incoming message. (In the current proposal this information is transmitted with each message. This is only necessary if we allow reception of a message by a process that does not know its complete data description.) A message capsule also contains message tag, context, control flags, and "matching" information. "Matching" information is used by a receiving process to select a message for receipt. "Matching" criteria has not been defined for MPI. A receiving process must also create and initialize a message capsule. When a message is received, the receive routine uses the information in the message capsule to move data from system buffers (or communication hardware) to location(s) in user space described in the capsule. The information in the message capsule is used by the receive routine to guide any data translation that may be necessary in a heterogeneous system. Note that the use of message capsules does not imply extra data movement. The message capsule just tells the system "Here's where to put the data when it arrives...". In the current proposal, message capsules are dynamic data objects. A message capsule can be reused once the data described by it is no longer required for message passing communication. Routines are provided to create and destroy message capsules, attach descriptions of user data, inquire about the contents of a message capsule, and modify message context. Some proposed routines are described below. The send and receive routines have "message capsule" as an additional parameter. Context is not required as an explicit input parameter to these routines. Proposed (incomplete) syntax for these routines is described below along with syntax for check, wait, probe, and release routines. Proposed syntax for "simplified" versions of send and receive routines are also described below. 2 Program Flow General steps for typical send -> receive communications using message capsules: Case 1. Receiving process knows the description of data in the message. 1) SENDING PROCESS: Sending process creates and initializes a message capsule by calling capsule creation and data description routines. Message context specification routine may also be called to select context if "default" context is not desired. A flag is set in the message capsule to indicate that it has been initialized by the user. If multiple messages with identical data descriptions are to be sent sequentially, this step only needs to be done for the first message. 2) RECEIVING PROCESS: Receiving process creates and initializes a message capsule by calling capsule creation and data description routines. Message context specification routine may also be called to select context if "default" context is not desired. A flag is set in the message capsule to indicate that it has been initialized by the user. If multiple messages with identical data descriptions are to be sent sequentially, this step only needs to be done for the first message. 3) SENDING PROCESS: "Send" message. During a send operation, the send (or wait for a non-blocking send) routine uses information in the message capsule to collect user data, construct a "message" (using system buffer(s) if necessary), and send the message to a specified destination. Depending on the implementation, the send routine may also perform data translation in a heterogeneous system. While a send operation is in progress, a flag is set in the capsule to indicate that the capsule should not be used for anything else. After the operation completes, this flag is cleared. A send routine will return an error if the message capsule has not been properly initialized. 4) RECEIVING PROCESS: "Receive" message. During a receive operation, the receive (or possibly wait for a non-blocking receive) routine distributes received data to user location(s) described in the message capsule. Depending on the implementation, the receive routine may also perform data translation in a heterogeneous system. While a receive operation is in progress, a flag is set in the capsule to indicate that the capsule should not be used for anything else. After the operation completes, this flag is cleared. The receive (or wait) routine will return an error if the data in the received message does not match the description in the capsule (ie. mismatch in data length(s) or type(s)). An error will also be returned if the message capsule has not been properly initialized. 5) RECEIVING PROCESS: (OPTIONAL) "Information" about message described by a capsule. Information about a message may be obtained by calling various "info" routines with the corresponding message capsule specified as an argument. Information returned by these routines includes data types, data sizes, data strides, sending process, message tag, and message context. "Info" routines will return an error if a message has not been successfully "probed" or received. Case 2. Receiving process does not know the description of data in the message. 1) SENDING PROCESS: Sending process creates and initializes a message capsule by calling capsule creation and data description routines (same as Case 1). 2) RECEIVING PROCESS: Receiving process creates a message capsule by calling the capsule creation routine. The message capsule may optionally be initialized in this step by calling the data description routines (though it may need to be re-initialized after the message arrives). Message context specification routine may also be called to select context if "default" context is not desired. 3) SENDING PROCESS: "Send" message. Same as Case 1. 4) RECEIVING PROCESS: (NOT OPTIONAL) "Probe" message (inquire if a message with "matching" characteristics has arrived-- "matching" criteria have not yet been defined for MPI). If a message with "matching" characteristics has arrived, data description information is loaded into a message capsule specified as an argument to the probe routine. The message is removed from the system message queue so no other thread may probe or receive the message unless it is "released" by the thread (see "release" in step 5 below). A flag is set in the message capsule to indicate that the data description of the arrived message is valid. This flag is used by the "info" routines. Note that the message capsule must be able to separately store the user's data description and the data description of the "arrived" message. "Info" routines will only return information about "arrived" messages. 5) RECEIVING PROCESS: (NOT OPTIONAL) "Info" routines MUST be called before a receive is posted when data description is not known by the receiving process. In a C program, information returned by these routines may be used to guide dynamic allocation of user data structures to store message data. In a Fortran77 program, this information can be used to check if available static arrays are large enough to hold message data before a receive is posted. 6) RECEIVING PROCESS: (OPTIONAL) "Release" message. If a thread does not wish to receive a message after a successful probe, it must release the message so other threads will have an opportunity to probe or receive the message. The release routine clears the flag mentioned in step 4 and puts the message back into the system message queue. Return to step 4. 7) RECEIVING PROCESS: Receiving process initializes the message capsule by calling data description routine(s) using information returned by the "info" routines. A flag is set in the message capsule to indicate that it has been initialized by the user. If the message capsule was initialized in step 2 and it does not have the same description as the arrived message, then the data description must be "reset" using a special routine, and then initialized. 8) RECEIVING PROCESS: "Receive" message. Same as Case 1. 3 Proposed Message Capsule Initialization and Inquiry Routines Basic functionality of proposed routines are described below. A sample syntax is shown just to make explanation easier. (For example, the issue of whether these routines are functions or subroutines is not important at this point, parameter ordering has not been given any thought, etc.) Also, routine names have been chosen to be descriptive an do not conform to any string length limitations. 3.1 Message Capsule Creation and Destruction Routines MPI_CREATE_CAPSULE (CAPSULE) Create message capsule CAPSULE. MPI_DESTROY_CAPSULE (CAPSULE) Destroy message capsule CAPSULE. 3.2 Data Description Routines Each message capsule contains one or more "data items" to describe the data in the corresponding message. Each data item contains information about data type, data length, and storage location(s). Each data item is created by a single call to a MPI_ATTACH_xxx() routine. Two kinds of data items are supported: "vector" and "strided". A vector data item is used for user data stored in contiguous memory. A strided data item is used for non-contiguous user data that can be described as a sequence of identically sized data blocks separated by a fixed stride. More general user data arrangements may be described by multiple vector and strided data items. (Note that the vector data item is a subset of the strided data item. Vector data items could be removed to get rid of almost half of the MPI_ATTACH_xxx() routines. We included the vector data item because it simplifies the most commonly used data description.) MPI_ATTACH_INT_VECTOR (CAPSULE, LENGTH, ARRAY) The next data item in the message described by message capsule CAPSULE is LENGTH integers. Storage location is user array ARRAY. MPI_ATTACH_REAL_VECTOR (CAPSULE, LENGTH, ARRAY) The next data item in the message described by message capsule CAPSULE is LENGTH real numbers (floats) stored in user array ARRAY. (etc. for other data types)... MPI_ATTACH_INT_STRIDE (CAPSULE, NUM_BLOCKS, BLOCK_LENGTH, STRIDE, ARRAY) The next data item in the message described by message capsule CAPSULE is NUM_BLOCKS blocks of integers. Each block has BLOCK_LENGTH integers. Starting elements in each block are separated by STRIDE integers. Storage location of first block is user array ARRAY. MPI_ATTACH_REAL_STRIDE (CAPSULE, NUM_BLOCKS, BLOCK_LENGTH, STRIDE, ARRAY) The next data item in the message described by message capsule CAPSULE is NUM_BLOCKS blocks of real numbers (floats). Each block has BLOCK_LENGTH real numbers. Starting elements in each block are separated by STRIDE real numbers. Storage location of first block is user array ARRAY. (etc. for other data types)... MPI_ATTACH_RESET (CAPSULE) All data items are removed from message capsule CAPSULE. The flag that indicates that CAPSULE has been initialized by the user is cleared. This routine can be used in step 7 of Case 2 in Section 2 above to remove any existing data item(s) before re-initializing the message capsule. It must be used in this situation if the message capsule contains a user data description that does not match the data description of the arrived message. This routine is also used to discard data descriptions that are no longer needed allowing message capsules to be re-used. 3.3 Context Modification Routine MPI_SET_CONTEXT (CAPSULE, CONTEXT) The context for the message described by message capsule CAPSULE is set to context CONTEXT. 3.4 "Information" Routines These routines must be used when a receiving process does not know the data description of a message it wants to receive. Use of these routines in other cases is optional. In all cases, "length" refers to number of elements (integers, floats, ...) not number of bytes. All of these routines return an error if the specified message has not been successfully probed or received. Only information about messages that have actually arrived is returned. Note that no "info" routine is required for context since this must always be known by the receiving routine. MPI_INFO_NUM_DATA_ITEMS (CAPSULE) Returns the number of data items in the message. MPI_INFO_DATA_ITEM_DESCRIPTION (CAPSULE, ITEM_NUMBER) Returns a description of data item number ITEM_NUMBER in the message described by message capsule CAPSULE. Valid return values include INT_VECTOR_ITEM, REAL_VECTOR_ITEM, INT_STRIDE_ITEM, REAL_STRIDE_ITEM, etc. MPI_INFO_VECTOR_ITEM (CAPSULE, ITEM_NUMBER, LENGTH) LENGTH is set to the length of the vector data described by data item number ITEM_NUMBER in message capsule CAPSULE. An error is returned if the data item is not a vector data item. MPI_INFO_STRIDE_ITEM (CAPSULE, ITEM_NUMBER, NUM_BLOCKS, BLOCK_LENGTH) NUM_BLOCKS is set to the number of blocks in the strided data described by data item number ITEM_NUMBER in message capsule CAPSULE. BLOCK_LENGTH is set to the length of each block. An error is returned if the data item is not a strided data item. Note that stride is not returned since there is no requirement that strides used by the sending and receiving processes match. MPI_INFO_TAG (CAPSULE) Returns the message tag for the message described by message capsule CAPSULE. MPI_INFO_SENDER (CAPSULE) Returns the sending process for the message described by message capsule CAPSULE. 4 Proposed (Partial) Syntax for Send, Receive, Check, Wait, Probe, and Release Routines Syntax examples showing how message capsules affect send, receive, check, wait, probe, and release routines are shown below. Routine naming is for clarity only and does not conform to any of the MPI naming styles. 4.1 Send Syntaxes for the three flavors of send discussed at the Dallas meeting ("synchronous", "blocking", and "non-blocking") are shown below. (These three routines could be reduced to one if the "communication mode" parameter were used. It is also possible to embed even more information in the message capsule such as "communication mode", message tag, and "matching" criteria. We show examples for this case in Section 8.) MPI_SEND_SYNCH (CAPSULE, DESTINATION, TAG) Send the message described by message capsule CAPSULE to process DESTINATION using message tag TAG. This is a "synchronous" send. MPI_SEND_BLOCKING (CAPSULE, DESTINATION, TAG) Send the message described by message capsule CAPSULE to process DESTINATION using message tag TAG. This is a "blocking" send. MPI_SEND_NON_BLOCKING (CAPSULE, DESTINATION, TAG) Send the message described by message capsule CAPSULE to process DESTINATION using message tag TAG. This is a "non-blocking" send. 4.2 Receive Syntaxes for the two flavors of receive discussed at the Dallas meeting ("blocking" and "non-blocking") are shown below. (These two routines could be reduced to one if the "communication mode" parameter were used.) "Matching" criteria have been left undefined. MPI_RECV_BLOCKING (CAPSULE, "MATCHING CRITERIA") Receive the message with characteristics satisfying "MATCHING CRITERIA" and store message data in location(s) described in message capsule CAPSULE. This is a "blocking" receive. Error conditions may be returned if the user-provided data description in the message capsule is not consistent with the data description of the received message (message too long, wrong data types, etc.). MPI_RECV_NON_BLOCKING (CAPSULE, "MATCHING CRITERIA") Receive the message with characteristics satisfying "MATCHING CRITERIA" and store message data in location(s) described in message capsule CAPSULE. This is a "non-blocking" receive. Error conditions may be returned if the user-provided data description in the message capsule is not consistent with the data description of the received message (message too long, wrong data types, etc.). (This will only happen if this routine is called when a "matching" message has already arrived.) 4.3 Wait and Check The wait and check routines use the message capsule in a way that is very similar to Jim Cownie's proposal (option 5 in "Removing internal state"). These routines are used with the non-blocking send and receive routines. MPI_CHECK (CAPSULE) CALLED BY A SENDING PROCESS: Return value indicates whether a send operation on the message described by message capsule CAPSULE has completed or not. An error will be returned if a non-blocking send has not been previously posted for CAPSULE. If the send operation has completed, clear the flag in CAPSULE that indicates that communication is in progress. CALLED BY A RECEIVING PROCESS: Return value indicates whether a receive operation on the message described by message capsule CAPSULE has completed or not. An error will be returned if a non-blocking receive has not been previously posted for CAPSULE. If the receive operation has completed, error conditions may be returned if the user-provided data description in the message capsule is not consistent with the data description of the received message (message too long, wrong data types, etc.). If the receive operation has completed, clear the flag in CAPSULE that indicates that communication is in progress. (Depending on the implementation, user data locations described in CAPSULE may be loaded by this routine, or they may already be loaded by the time this routine is called.) MPI_WAIT (CAPSULE) Same as MPI_CHECK() except that the routine will not return until a "matching" message arrives. 4.4 Probe and Release The probe routines also uses the message capsule in a way that is very similar to Jim Cownie's proposal (option 5 in "Removing internal state"). Both wait and check styles of probe are proposed. MPI_PROBE_CHECK (CAPSULE, "MATCHING CRITERIA") If a message with characteristics satisfying "MATCHING CRITERIA" has arrived, copy its description into message capsule CAPSULE. The message is removed from the system "message queue" and may not be probed or received by another thread unless explicitly released using the release routine. "Information" routines can be called using CAPSULE to get information about the message. A receive can also be posted for this message by the thread that owns CAPSULE (after modifying CAPSULE using information from the "info" routines). If a "matching" message has not arrived, return immediately without modifying CAPSULE. Return value indicates whether a message has arrived or not. MPI_PROBE_WAIT (CAPSULE, "MATCHING CRITERIA") Same as MPI_PROBE_CHECK() except that the routine will not return until a "matching" message arrives. MPI_RELEASE (CAPSULE) The previously probed message described by message capsule CAPSULE is put back in the system "message queue". It may now be probed or received by any thread. An error will be returned if the message has not been probed. Data description of arrived message in CAPSULE is invalidated. 4.5 Simplified Versions of Send and Receive Routines Good arguments have been made (we think) for "simplified" versions of send and receive routines that are less flexible and "easier to use". These routines could be built on top of the routines proposed above or implemented directly. The simplified routines proposed below cannot be used with contexts or probe or "info" routines. Only contiguous user data is supported. Receiving process must know message data description. Actually, it is not clear to us that these routines are really easier to use. They do conform more closely to "common practice" though. Simplified versions of the check and wait routines can also be made (we've left them out-- this thing is long enough already!). MPI_SEND_SYNCH_SIMPLE (DESTINATION, TAG, ARRAY, LENGTH, DATA_TYPE) Send message consisting of LENGTH elements from location ARRAY to process DESTINATION using message tag TAG. DATA_TYPE is the data type for ARRAY. (Pre-defined values for DATA_TYPE are MPI_INT, MPI_REAL, etc. DATA_TYPE is required for heterogeneous communication.) This is a "synchronous" send. MPI_SEND_BLOCKING_SIMPLE (DESTINATION, TAG, ARRAY, LENGTH, DATA_TYPE) Same as MPI_SEND_SYNCH_SIMPLE() except that this is a "blocking" send. MPI_SEND_NON_BLOCKING_SIMPLE (DESTINATION, TAG, ARRAY, LENGTH, DATA_TYPE) Same as MPI_SEND_SYNCH_SIMPLE() except that this is a "non-blocking" send. MPI_RECV_BLOCKING_SIMPLE ("MATCHING CRITERIA", ARRAY, LENGTH, DATA_TYPE) Receive the message with characteristics satisfying "MATCHING CRITERIA" and store message data in array ARRAY. Received data type must be DATA_TYPE. Data length must be LENGTH. This is a "blocking" receive. Error conditions will be returned if LENGTH or DATA_TYPE are not consistent with the content of the received message. MPI_RECV_NON_BLOCKING_SIMPLE ("MATCHING CRITERIA", ARRAY, LENGTH, DATA_TYPE) Same as MPI_RECV_BLOCKING_SIMPLE() except that this is a "non-blocking" receive. 5 Code Fragment Examples in "C" Here are some code fragment examples of how programs using message capsules might look. Lots of stuff has been left out (some variable declarations and initializations, error control using return values, etc.) for brevity. Code segments for sending and receiving processes are shown separately for clarity. "Matching" criteria for receives is left undefined. Process ID is assumed to be an integer. 5.1 Blocking Send -> Receive Vector Data Item: Receiver Knows Data Description DECLARATIONS FOR SENDING AND RECEIVING PROCESSES: Capsule *capsule; /* pointer to message capsule */ int i, tag, receiver_pid; float my_vector[1000]; /* user data */ SENDING PROCESS CODE: ... MPI_CREATE_CAPSULE(capsule); /* create message capsule */ ... MPI_ATTACH_REAL_VECTOR(capsule, 1000, my_vector); /* initialize capsule */ ... for (i=0; i<10; i++) /* loop 10 times */ { ... compute(my_vector, ...); /* user does something to data */ MPI_SEND_BLOCKING(capsule, receiver_pid, tag); /* send msg to receiver */ ... } /* end of i for loop */ ... MPI_DESTROY_CAPSULE(capsule); /* destroy capsule before end of program */ ... RECEIVING PROCESS CODE: ... MPI_CREATE_CAPSULE(capsule); /* create message capsule */ ... MPI_ATTACH_REAL_VECTOR(capsule, 1000, my_vector); /* initialize capsule */ ... for (i=0; i<10; i++) /* loop 10 times */ { ... MPI_RECV_BLOCKING(capsule, "MATCHING CRITERIA"); /* receive message */ mess_with(my_vector, ...); /* user does something with data */ ... } /* end of i for loop */ ... MPI_DESTROY_CAPSULE(capsule); /* destroy capsule before end of program */ ... 5.2 Blocking Send -> Receive Vector Data Item: Receiver Does Not Know Data Description DECLARATIONS FOR SENDING AND RECEIVING PROCESSES: Capsule *capsule; /* pointer to message capsule */ int tag, receiver_pid, vec_length, num_items, item, data_item; float my_vector[1000]; /* user data */ SENDING PROCESS CODE: ... MPI_CREATE_CAPSULE(capsule); /* create message capsule */ ... MPI_ATTACH_REAL_VECTOR(capsule, 1000, my_vector); /* initialize capsule */ ... MPI_SEND_BLOCKING(capsule, receiver_pid, tag); /* send msg to receiver */ ... MPI_DESTROY_CAPSULE(capsule); /* destroy capsule before end of program */ ... RECEIVING PROCESS CODE: ... MPI_CREATE_CAPSULE(capsule); /* create message capsule */ ... MPI_PROBE_WAIT(capsule, "MATCHING CRITERIA"); /* probe until message arrives*/ /* Get data description of arrived message using "info" routines and */ /* initialize message capsule. */ num_items = MPI_INFO_NUM_DATA_ITEMS(capsule); /* get number of data items */ /* (Resetting capsule is optional if this is the only place this occurs in */ /* the code. Resetting is necessary if this code fragment (excluding */ /* capsule creation and destruction) is in a loop. Resetting does not */ /* affect information contained in the capsule about the arrived message.) */ MPI_ATTACH_RESET(capsule); if (num_items == 1) /* Deal with single data item message. */ { /* What kind of data item is in the message? */ data_item = MPI_INFO_DATA_ITEM_DESCRIPTION(capsule, 0); if (data_item == REAL_VECTOR_ITEM) /* Deal with real vector data item. */ { /* Get length of vector item and store in "vec_length". */ MPI_INFO_VECTOR_ITEM (capsule, 0, &vec_length); if (vec_length <= 1000) /* Data fits in existing user array. */ { /* Initialize first data item to be "vec_length" floats stored */ /* at location "my_vector". */ MPI_ATTACH_REAL_VECTOR(capsule, vec_length, my_vector); } /* end of vec_length if */ else { ... /* allocate a larger buffer to hold message data */ } /* end of vec_length else */ } /* end of data_item if */ else { ... /* deal with other possibilities for first data item */ } /* end of data_item else */ } /* end of num_items if */ else { ... /* deal with multiple data items */ } /* end of num_items if */ MPI_RECV_BLOCKING(capsule, "MATCHING CRITERIA"); /* receive message */ ... MPI_DESTROY_CAPSULE(capsule); /* destroy capsule before end of program */ ... 5.3 Blocking Send -> Non-Blocking Receive Complicated Message With Context: Receiver Knows Data Description DECLARATIONS FOR SENDING AND RECEIVING PROCESSES: Capsule *capsule; /* pointer to message capsule */ int i, tag, receiver_pid, my_context; float my_vector[1000], my_array[2000]; /* user data */ int my_indices[200]; /* user data */ SENDING PROCESS CODE: ... MPI_CREATE_CAPSULE(capsule); /* create message capsule */ ... MPI_SET_CONTEXT (capsule, my_context); /* my_context must be set up earlier */ MPI_ATTACH_REAL_VECTOR(capsule, 1000, my_vector); /* initialize capsule */ MPI_ATTACH_REAL_STRIDE (capsule, 10, 20, 200, my_array); MPI_ATTACH_INT_VECTOR(capsule, 200, my_indices); ... MPI_SEND_BLOCKING(capsule, receiver_pid, tag); /* send msg to receiver */ ... MPI_DESTROY_CAPSULE(capsule); /* destroy capsule before end of program */ ... RECEIVING PROCESS CODE: ... MPI_CREATE_CAPSULE(capsule); /* create message capsule */ ... MPI_SET_CONTEXT (capsule, my_context); /* must be same as sender's context */ MPI_ATTACH_REAL_VECTOR(capsule, 1000, my_vector); /* initialize capsule */ MPI_ATTACH_REAL_STRIDE (capsule, 10, 20, 200, my_array); MPI_ATTACH_INT_VECTOR(capsule, 200, my_indices); ... MPI_RECV_NON_BLOCKING(capsule, "MATCHING CRITERIA"); /* post receive */ ... /* "do something" */ MPI_WAIT (capsule); /* wait for message when done */ ... MPI_DESTROY_CAPSULE(capsule); /* destroy capsule before end of program */ ... 5.4 Blocking Send -> Receive With "Simplified" Routines DECLARATIONS FOR SENDING AND RECEIVING PROCESSES: int i, tag, receiver_pid; float my_vector[1000]; /* user data */ SENDING PROCESS CODE: ... for (i=0; i<10; i++) /* loop 10 times */ { ... compute(my_vector, ...); /* user does something to data */ /* send msg to receiver */ MPI_SEND_BLOCKING_SIMPLE (receiver_pid, tag, my_vector, 1000, MPI_REAL); ... } /* end of i for loop */ ... RECEIVING PROCESS CODE: ... for (i=0; i<10; i++) /* loop 10 times */ { ... /* receive message */ MPI_RECV_BLOCKING_SIMPLE ("MATCHING CRITERIA", my_vector, 1000, MPI_REAL); mess_with(my_vector, ...); /* user does something with data */ ... } /* end of i for loop */ ... 6 Advantages and Disadvantages of "Message Capsule" Approach 6.1 Advantages - The number of separate send and receive routines is greatly reduced without sacrificing functionality. - A user who is used to "common practice" can use the simplified versions of the routines. - A user who wants more flexibility only needs to learn about the features required for his/her specific application. (For example, if I only need contiguous messages, then I don't need to know anything about strided data items. If the receiving process always knows the data description of received messages, then I don't need to know about the probe and "info" routines.) - "Hidden states" are removed so multi-threaded applications won't get confused. - Encapsulation of features in message capsules allows new features to be added later without modifying syntax of existing routines. A new feature would require addition of one or more new routines to modify and examine message capsules. 6.2 Disadvantages - The use of message capsules does not conform exactly to "common practice". It's fairly close to PVM though. - There is no way to prevent a user from messing with a message capsule directly. This is the same problem encountered in C with (FILE *) structures. The problem can be partially alleviated by including "magic numbers" in message capsules. - Possible slight performance degradation due to "casing" in receive routines. - Data description must be sent along with each message. This is only a significant problem if lots of short messages are sent by a program running on a machine with low communication latency. Sending data description information could be avoided if we do not allow a receiving process to be ignorant of the data description of incoming messages. 7 Unresolved Issues - Can the message capsule concept work well with collective communication routines? - Is it really necessary to support multiple data items in a message capsule? It is if we want to allow mixed-type messages. 8 Code Fragment Example With Even More Information Hidden In the Message Capsule The code fragment presented in Section 5.1 has been modified slightly to indicate how the capsule could possibly contain additional information to further reduce the number of send/receive calls and the number of parameters. Message tag and "matching" criteria for receive operations are now "hidden" in the message capsule. Communication "mode" (synchronous, blocking, or non-blocking) is used to reduce the number of send and receive routines. Communication "mode" is also "hidden" in the message capsule. Four new routines have been added. Routine MPI_ASSIGN_TAG() sets message tag. Routines MPI_SET_SEND_MODE() and MPI_SET_RECV_MODE() set communication "mode" (synchronous, blocking, or non-blocking for send, blocking or non-blocking for receive). Routine MPI_SET_MATCH() sets matching characteristics for receive operations. The send routines (MPI_SEND_SYNCH(), MPI_SEND_BLOCKING(), and MPI_SEND_BLOCKING()) have been combined into one routine (MPI_SEND()). The receive routines (MPI_RECV_BLOCKING() and MPI_RECV_NON_BLOCKING()) have been combined into one routine (MPI_RECV()). Both MPI_SEND() and MPI_RECV() have a reduced number of parameters. In the example below, lines containing "###" indicate changes from the example in Section 5.1. (Example from 5.1) Blocking Send -> Receive Vector Data Item: Receiver Knows Data Description DECLARATIONS FOR SENDING AND RECEIVING PROCESSES: Capsule *capsule; /* pointer to message capsule */ int i, tag, receiver_pid; float my_vector[1000]; /* user data */ SENDING PROCESS CODE: ... MPI_CREATE_CAPSULE(capsule); /* create message capsule */ ... /* ### Following 4 lines added ### */ MPI_ASSIGN_TAG (capsule, tag); /* Assign the tag to the capsule */ /* Set send mode to "blocking". Valid values for send mode are */ /* MPI_MODE_SEND_NON_BLOCK, MPI_MODE_SEND_BLOCK, and MPI_MODE_SEND_SYNC. */ MPI_SET_SEND_MODE (capsule, MPI_MODE_SEND_BLOCK); MPI_ATTACH_REAL_VECTOR(capsule, 1000, my_vector); /* initialize capsule */ ... for (i=0; i<10; i++) /* loop 10 times */ { ... compute(my_vector, ...); /* user does something to data */ /* ### tag parameter disappears from parameter list ### */ MPI_SEND(capsule, receiver_pid); /* send msg to receiver */ ... } /* end of i for loop */ ... MPI_DESTROY_CAPSULE(capsule); /* destroy capsule before end of program */ ... RECEIVING PROCESS CODE: ... MPI_CREATE_CAPSULE(capsule); /* create message capsule */ ... /* ### Following 5 lines added ### */ /* Set matching criteria in capsule */ MPI_ASSIGN_MATCH(capsule, "MATCHING CRITERIA"); /* Set the receive mode to "blocking". Valid values for receive mode are */ /* MPI_MODE_RECV_NONBLOCK and MPI_MODE_RECV_BLOCK. */ MPI_SET_RECV_MODE (capsule, MPI_MODE_RECV_BLOCK); MPI_ATTACH_REAL_VECTOR(capsule, 1000, my_vector); /* initialize capsule */ ... for (i=0; i<10; i++) /* loop 10 times */ { ... /* ### tag parameter disappears from parameter list ### */ MPI_RECV(capsule); /* receive message */ mess_with(my_vector, ...); /* user does something with data */ ... } /* end of i for loop */ ... MPI_DESTROY_CAPSULE(capsule); /* destroy capsule before end of program */ ... From owner-mpi-pt2pt@CS.UTK.EDU Sat Jan 16 05:16:45 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA00760; Sat, 16 Jan 93 05:16:45 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA24719; Sat, 16 Jan 93 05:16:13 -0500 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Sat, 16 Jan 1993 05:16:08 EST Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from gatekeeper.oracle.com by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA24593; Sat, 16 Jan 93 05:15:56 -0500 Received: from jewel.us.oracle.com by gatekeeper.oracle.com (5.59.11/37.7) id AA21412; Sat, 16 Jan 93 02:15:53 PST Received: by jewel.us.oracle.com (5.59.10/37.3) id AA01226; Sat, 16 Jan 93 02:21:14 PST Message-Id: <9301161021.AA01226@jewel.us.oracle.com> Date: Sat, 16 Jan 93 02:21:14 PST From: Charles Simmons To: mpi-pt2pt@cs.utk.edu Subject: mpi Dear Sirs, Over the past 4 years, some of us at Oracle have been addressing many of the same problems that you are addressing in your MPI effort. We have approached the problem from a C/Unix/Systems Software point of view instead of a Fortran/Scientific Software point of view. We learned of your effort from Sandia, whom we are hoping to work with in developing systems software for MP machines, and the communications interface will be a substantial part of this effort. I sent the following comments about MPI to Jack Dongarra. He suggested dropping the note on the 'pt2pt' mailing list. My primary hope is that my radically different background and point of view will stimulate new ideas. My expectations, however, are that you will find me obnoxious and ask me to go away. :-) Cheers, Chuck Simmons csimmons@oracle.com General Comments ---------------- The proposed interface specification is at best disappointing. From our point of view, it essentially standardizes and extends the existing nCUBE model of a message passing interface. We explicitly rejected this model two years ago. In this document, I'll describe why I think this interface model is inadequate. The most important aspect of the proposed standard is its lack of reference to prior related work. Communications interfaces have been standardized before. Thus, the most important question that the proposed standard must address is: Why does no existing interface meet the requirements of our applications? Associated with this are the questions: What concepts can we steal from existing interfaces? What can we learn from past mistakes? The current work on the MPI interface appears to be occuring in a vaccuum, ignoring prior work. The introduction to the draft standard states: "The main advantages of establishing a message passing standard are portability and ease-of-use." If that is the case, then the MPI effort is seriously misguided. For example, the existing TCP/IP standard is far more portable and easier to use than MPI will ever be. However, I would agree that TCP/IP is not adequate as a communications interface for an MP machine. The reason for this is that MP machines provide relatively high-bandwidth low-latency communications. To take advantage of this, a very lightweight communications protocol is required. We first need a highly efficient communications interface; we then need a portable and easy-to-use interface. An efficient communications interface needs to provide the following properties: (1) We require the ability to copy messages from a user address space directly onto a wire without an intervening memcpy() of the message into a system buffer. Similarily, we require that received messages be able to move from a wire into a user address space without an intervening memcpy(). (2) We require that messages be packetized. That is, in TCP/IP, individual message packets are concatenated together at the sender and sent out as a stream of data. This requires the receiver to parse the incoming data stream to recover the individual packets. In an efficient interface, we want the packet boundaries to be preserved. Once we have outlined our requirements from the communications interface, the next question to ask is whether we are ready to standardize on an approach. Standards bodies work best when they standardize existing work, and work worst when they attempt to design. Standardizing too early on an inferior interface will impede research efforts to improve the interface, and additional work will be needed when it becomes time to standardize the desired interface. If it is decided that a standard is desired, then the scope of the standard needs to be carefully delineated. A reasonable scope is that of defining a communications interface at about the ISO network or transport layer. Unfortunately, some of the text in the draft standard suggest that some of the researchers want to address all of the issues associated with building an MP operating system. For example, section 5 refers to creating and destroying processes and to addressing distributed computing in a heterogeneous environment. Appendix A contains routines that are completely unrelated to communications, and which may well conflict with other existing standards, or which are already well defined in other existing standards. (Consider, for example, the date and time routines. Compare and constrast these to functionality provided by ANSI C and/or Posix.) In general, most of the draft standard stays within a reasonable scope. Attempts to extend the standard beyond that reasonable scope should be brutally (:-) suppressed. The core issues of the standard are: 1) Point-to-point communications. This is the heart and soul of the standard. 2) Communications amongst multiple instantiations of a single program (a group or collection), and communications between these collections. This is a reasonable topic to explore since these collections will be common entities on MP machines. However, this is far less important than point-to-point communications. Given a good portable interface for point-to-point communications, an application can fairly easily implement completely portable and reasonably efficient inter- and intra- collection communications. 3) Reliable messaging and sequencing of messages. This is, for the most part, explicitly ignored by the draft standard. However, this is far more important than (2). As mentioned before, applications don't need (2) to be standardized in order to be portable. However, it is extremely difficult to write the code needed to provide robust, efficient reliable messaging. 4) Threads (aka "Communications Contexts"). For the most part, this is orthogonal to communications. Threads need to be addressed to the extent that the communications interface model must be capable of coexisting with a reasonable threads implementation. In summary, the draft standard should contain a section describing the scope of the standard. Issues such as heterogeneity, performance tracing, parallel I/O, process creation, load balancing, and environmental queries should be placed explicitly outside the scope of the standard, except to the extent that the standard needs to coexist with these features. [Note: On page 3, a reference is made to dynamic load balancing. The interesting problem of dynamic load balancing from the communications interface point of view is that some addresses known to some processes may become invalid at arbitrary points in time. The operating system may need to be able to forward messages that were sent to an obsolete address.] Specific Comments ----------------- My specific objections to details of the MPI standard fall into the following areas: 1) Reliability and sequencing. I feel that the standard should pay far more attention to these subjects. Reliability and sequencing of messages are extremely important in industrial strength applications. Because of the importance of this area and its difficulty of implementation, the standard should pressure hardware vendors and operating system vendors to provide this functionality. On page 6 of the draft standard, for example, it is explicitly allowed that an operating system may discard a message if that message would overflow system buffers, and the operating system need not inform the sender that the message has been discarded. This immediately requires most applications to make use of inefficient algorithms to detect when a message may have been discarded and retry the send of the message as well as to detect when a message has been successfully received but also retried. This is so onerous, that we have asked nCUBE to implement hardware for their next generation system so that when an OS receives a message, it can immediately return one of the following stati to the sending OS: message accepted; message tossed due to parity error; message tossed due to lack of space. Allowing unreliable messages to be implemented encourages the implementation of unreliable applications. 2) The mechanisms used to receive a message are inefficient. In particular, many applications make use of variable length messages. The current proposal only works well for fixed length messages. In the face of variable length messages, the current proposal requires that an application either over allocate storage; or it requires that the application allow the OS to buffer the message, and then the application will probe for the message to determine the length, allocate a correctly sized buffer, and copy the message into the new buffer. Essentially, we want the OS to be able to copy messages off of a wire directly into user address space, so that it is then easy for the application to access variable length packets. We are willing to accept interfaces that require the hardware to implement paging for efficient implementation. 3) The 'mode' argument on the send and receive functions is inefficient. We strongly suspect that this argument will virtually always be a constant. If that is the case, it is more appropriate to make this constant part of the function name instead of an argument. For example, the obvious implementation of the mpi_csend() routine would look like: int mpi_csend (mode, buf, dest, type, len) ... { switch (mode) { case NON_BLOCKING: return mpi_csend_blocking (buf, dest, type, len); case BLOCKING: return mpi_csend_nonblocking (buf, dest, type, len); case SYNCHRONOUS: return mpi_csend_synchronous (buf, dest, type, len); default: return (save_errno, -1); } } static int mpi_csend_nonblocking (buf, dest, type, len) ... { trap to os; save any error; return value; } static int mpi_csend_blocking (buf, dest, type, len) ... { int msgid; msgid = mpi_csend_nonblocking (buf, dest, type, len); mpi_wait (msgid); return status/length sent; } ... The costs of this approach are: (A) An extraneous error condition when the value of 'mode' is out of bounds. Making the mode part of the function name allows this error to be detected at compile time. (B) Additional cpu cycles wasted passing an additional argument to various routines. (C) Cpu cycles as we decide which piece of code to execute in response to the 'mode'. Further, we take the point of view that the non-blocking operations are of fundamental importance, and that the other modes of operation are relatively minor, uninteresting envelopes around this fundamental operation. 4) The send and receive routines for scatter-gather messages ignore prior work. Consider, for example, the closely related Unix readv() and writev() operations. These are specified as: int readv (int fd, struct iovec *iov, int iovcnt); Since an 'iovec' structure is already defined, this structure should be used instead of inventing a new paradigm that separates this into two distinct arrays. In particular, the draft standard pretty much requires system software to perform an expensive translation to convert iovec's into separate arrays. [Note also that we consider the sending of a contiguous message as a relatively uninteresting envelope around the more fundamental transmission of a scatter-gather message.] 5) The send and receive routines specify the use of a pair of arguments. At a conceptual level, these arguments are a single concept: the address to which a message should be sent (or from which the message should be received). This approach has numerous problems: A) Passing two arguments to specify an address is slower than passing a single argument. B) This implies an implementation whereby each process has one message receive queue. To pull messages off this queue in an order other than that in which the message was received requires a potentially time-consuming queue scan to search for a desired message. A much better approach would allow the OS to take a received message and quickly hash it to one of a few receive queues for a process. Since the code that sends a message is generally closely related to the code that receives a message, the sender can make sure that it puts the message on a queue in such a fashion that the receiver will never be interested in anything other than the head of one of its queues. C) Allocation of message type constants by library code in a robust fashion is exceedingly difficult. If message types are to be required, the standard must address the mechanisms that are to be used to allocate message types so that separately developed libraries are guaranteed to have no conflicts. We prefer the concept of a "port". Essentially, this allows a process to ask the OS to create another receive queue for the process. Sending processes then send messages to a specific queue. A receiving process can retrieve a message from the head of a specific queue, or it can use a routine similar to the Unix select() system call to provide a bitmap of interesting queues and retrieve a message from the head of one of these queues. Taking the above into account, we would then specify the interface as: /* non-blocking scatter/gather recv */ int mpi_recv (port_t local_port, struct iovec *iovecp, int iovec_len); /* blocking version of the above */ int mpi_recvb (port_t local_port, struct iovec *iovecp, int iovec_len); /* synchronous version of the above */ int mpi_recvs (port_t local_port, struct iovec *iovecp, int iovec_len); /* vector versions of the above */ int mpi_recvv (port_t local_port, void *bufp, int blklen, int stride, int nblks); int mpi_recvvb (port_t local_port, void *bufp, int blklen, int stride, int nblks); int mpi_recvvs (port_t local_port, void *bufp, int blklen, int stride, int nblks); /* contiguous versions of the above */ int mpi_recvc (port_t local_port, void *bufp, int buflen); int mpi_recvcb (port_t local_port, void *bufp, int buflen); int mpi_recvcs (port_t local_port, void *bufp, int buflen); Also, in the above, the vector versions of the above routines may need additional work. We envision an rpc protocol whereby an rpc header is prepended to a vector that is to be scattered. The application would like to receive the rpc header into one buffer and then scatter the vector into numerous locations. The most general approach would be for the primitive operation to specify a list of arrays into which the received buffer is to be vector scattered. For mpi_infos() and mpi_infot()... Here you are addressing the problem of how to handle an OS header that is prepended to every message. We agree that the general problem is somewhat tricky. Which is probably why the proposed implementation has some constraints on it that are less than trivially easy to use. Note that in a C program running on Unix, the application may need to disable signals before it calls a recv routine, and re-enable the signals after calling a routine to retrieve header information. One question that needs to be answered is the purpose of this header information. If the purpose is application specific, then perhaps the application should have embedded this information in its RPC header. (Our most generally used RPC protocol does this. This approach also allows a thread context pointer to be easily embedded in the message and appropriately retrieved.) If the purpose of this header is for security and verification that a sender has permission to perform an operation, then the simple source/type information available here is probably inadequate. (The existing Unix paradigm is to provide a cred_t structure in this header that specifies the credentials of the sender. This header information is sufficiently burdensome that an application may want to tell the operating system whether or not to attach credentials information to each message.) For mpi_probe(), we strongly desired both blocking and non-blocking versions of this. The non-blocking version simply tests for the presence of a message and returns. The blocking version blocks until a desirable message is present. Consider again the select() system call which provides for both of these functionalities as well as the ability to specify a timeout on the blocking version of the operation. I'm not qualified to comment on groups since we don't use those much, and when we do, we completely ignore little things like topology and locality. However, lack of qualification never stopped me before... We've been toying with the idea of an "array port". The concept is not well defined. For certain types of applications, we may be able to create a port in a number of processes such that the ports have the property that the bit pattern of one port is easily computable from the bit pattern of another port. The primary goal here is to use a very small amount of storage in each process to specify the addresses of all processes. I would appreciate a lot more information about the intended use of groups. Certainly the proposal is far more complicated than anything I would ever use. Also, it wouldn't work for things that I would use. In particular, when I use a group, the first thing that I am doing is creating a list of addresses for all processes in the group. I cannot implement this using MPI_DEFRG. In particular, the processes in the group don't know the addresses of the processes in the group until after all processes have become part of the group. Typically, what we do is have one process in the group store its address in a well known location (a nameserver). Other processes in the group wait for the first process to appear, and then they send their addresses to the first process. The first process generates the list of addresses and broadcasts it back to all of the processes (now that all the processes are known). Thus, the list of addresses is our canonical data structure. Our canonical operations are the routines that help create this data structure, the routines that help us broadcast the data structure, and the routines that allow us to communicate pt-to-pt to an indexed address in the data structure. This approach is easily extended to allow processes outside the group to communicate with processes inside the group. The mpi_*unpack routines specify that the 'msg' argument is an output argument. It should really be an input argument. For the general pack/unpack routines; again, using a 'struct iovec *' instead of would prevent conflicts with existing paradigms. For section A.4, this section is completely outside the scope of the document. The communications contexts essentially implement threads. The communications routines should not specify how you get information about threads, nor how you go about creating a new thread. Also, the PUSH and POP approach to communications and group contexts looks like a design flaw waiting to bite. In section A.5, the cpu,date,machine,infmn,time, and wall routines are outside the scope of the document. The date and time functions are particularly suspect. Compare then to the ANSI C and Posix routines for manipulating the date and time. The mpi_etext and mpi_error routines are reasonable. Note that in our heavily layered applications, we will constantly need to convert mpi_error codes into Unix error codes, and it might be nice if the MPI error handling paradigm was better integrated into Posix error handling. (All we care is that in addition to implementing mpi_error and mpi_etext, the implementation set 'errno' to a reasonable value.) Note that it is not at all clear that communications contexts will work except for in highly restricted circumstances. They are less general than either threads or ports. In fact, I suspect that ports can provide all the needed functionality of communications contexts, and message types, and pids with less hassle. From owner-mpi-pt2pt@CS.UTK.EDU Sat Jan 16 06:04:17 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA11435; Sat, 16 Jan 93 06:04:17 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA00620; Sat, 16 Jan 93 06:03:55 -0500 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Sat, 16 Jan 1993 06:03:46 EST Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from gatekeeper.oracle.com by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA00612; Sat, 16 Jan 93 06:03:37 -0500 Received: from jewel.us.oracle.com by gatekeeper.oracle.com (5.59.11/37.7) id AA21711; Sat, 16 Jan 93 03:03:35 PST Received: by jewel.us.oracle.com (5.59.10/37.3) id AA01231; Sat, 16 Jan 93 03:08:56 PST Message-Id: <9301161108.AA01231@jewel.us.oracle.com> Date: Sat, 16 Jan 93 03:08:56 PST From: Charles Simmons To: mpi-pt2pt@cs.utk.edu Subject: RE: Communication contexts > | 4) Chuck Simmons of Oracle has suggested that communication contexts > | are really a particular type of thread, and so can be handled > | using existing threads packages. > possible. 4) is a problem on systems that don't support threads, and we have > more-or-less agreed to be consistent with thread packages but not depend upon > them. 3) should be discussed along with the converse: that contexts rather Allow me to clarify my sentiments. The example of context usage given in the MPI draft documentation appears to explicitly assume the existance of threads, and here contexts appear to be being used as threads. > 5) Contexts are used exclusively to insure that message collisions will not > occur if independently developed sub-programs are combined. Contexts and > groups are orthogonal. Contexts and threads are orthogonal. Each message > has an associated context and tag. Message context is managed by library > routines and is completely out of a user's control. Message tag is > selected by the user. I could strongly support almost all of the above. [The part I wouldn't support is a "tag" as I will explain below.] We do our work on the nCUBE, which, as you may know, has a message passing interface essentially equivalent to that proposed by MPI. We ran into the exact problem quoted above: it was difficult to write libraries that we're guaranteed not to conflict in their usage of message types. The usage of the word "contexts" in the above is equivalent to the usage of the word "ports" amongst operating systems programmers. Because "port" is slightly less overloaded than "context", I prefer the word "port". Note that ports are sufficiently powerful that they subsume the concepts of "contexts", "pids", and "message types" or "tags" as used by MPI. >The only expression stated so far about the difference between tags and >contexts I had gleaned was that context should now be wildcardable (IE no >-1 for All contexts) while a receive ALL or some other MASK variant would >be allowed on tags. I agree that there is little difference between tags and contexts. This helps explain why ports are so good at subsuming both contexts and tags. > The method of managing message contexts is a separate issue (assuming we > want contexts). Existing proposals are: > > 5a) Stack-based management (objected to due to hidden states). > 5b) Explicit registration with user-defined "names" (probably requires > some communication). > 5c) Explicit registration by a central authority ("dollar bill" > registration mentioned by Jim Cownie.) I don't particularly understand the above, probably because I haven't been listening in on enough of the discussion. I think I understand what you mean by stack-based management, and we do use the word "registration"... When using ports, stack-based management isn't an issue for the same reasons you don't manage pids and message types using stack based management. For example, when sending a message, instead of using the MPI push_context (contextp); send (dest, type, buf, buflen); pop_context (contextp); you use send (port, buf, buflen); In our usage, a "port" is simply a queue to which messages can be sent. The receiver can remove a message from the head of the queue. Wildcards are neither needed nor implemented, making ports more efficient. The efficiency arises from the fact that a process can have multiple ports. In MPI, you essentially implement one receive queue for each pid. If you want to use a wildcard to access messages out-of-order, you need to waste time scanning the queue. With multiple ports, however, you can set up multiple queues so that you never need to access anything other than the head of a queue. Sometimes ports do need to be "registered". In our terminology, registration is the process of publishing in a well-known location sufficient information about a "port" created by one process that another process can send messages to that port. This is a very high level action that is almost outside the scope of MPI. We use a nameserver for this purpose. (The nameserver is accessed via a well-known port, i.e. a port whose bit-pattern is a well-known constant.) [Do note that we require that the bit-pattern representing a port be allowed to be transmitted between to processes without the operating system needing to know that the bit-pattern represents a port. This means that accessing the nameserver is not required in order to use a port.] For groups, we have thought about extending the concept of a port to an array-port. The simple implementation of this allows each process in a group to create a port. These ports are all sent to a central location which creates an array of ports and broadcasts the array to all ports in the array. The complex memory efficient implementation may require processes of a group to be created in a special fashion. Basically, it allows the address of a port for a specific process in a group to be easily computed by another process given the port of the first process in the group and the index of the target process in the group. -- Chuck From owner-mpi-pt2pt@CS.UTK.EDU Tue Jan 19 11:38:29 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA17983; Tue, 19 Jan 93 11:38:29 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA21908; Tue, 19 Jan 93 11:38:02 -0500 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Tue, 19 Jan 1993 11:37:57 EST Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from marge.meiko.com by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA21879; Tue, 19 Jan 93 11:37:21 -0500 Received: from hub.meiko.co.uk by marge.meiko.com with SMTP id AA13277 (5.65c/IDA-1.4.4); Tue, 19 Jan 1993 11:36:49 -0500 Received: from float.co.uk (float.meiko.co.uk) by hub.meiko.co.uk (4.1/SMI-4.1) id AA27411; Tue, 19 Jan 93 16:36:38 GMT Date: Tue, 19 Jan 93 16:36:37 GMT From: jim@meiko.co.uk (James Cownie) Message-Id: <9301191636.AA27411@hub.meiko.co.uk> Received: by float.co.uk (5.0/SMI-SVR4) id AA05440; Tue, 19 Jan 93 16:35:18 GMT To: csimmons@us.oracle.com Cc: mpi-pt2pt@cs.utk.edu, mpi-context@cs.utk.edu In-Reply-To: Charles Simmons's message of Sat, 16 Jan 93 02:21:14 PST <9301161021.AA01226@jewel.us.oracle.com> Subject: mpi Content-Length: 3447 To: mpi-pt2pt@meiko.co.uk, mpi-context.@meiko.co.uk (Apologies to those who get it twice) Gentlepeople, Chuck Simmons has raised some interesting issues, with which I have a great deal of sympathy. The original messaging model which we implemented on our machines (and which we still support, and expect to continue to support) is in exactly the vein which Chuck is asking for (and has been used for an Oracle port !). In particular CSN supports 1) multiple end points for communication in a single process (know as "transports") 2) No tagging of messages 3) No system buffering, but the ability for users to queue multiple non-blocking receives on a transport, thus providing the buffering they require. 4) Send and receive by struct iovec. (As Chuck observes, the implementation ends up building an iovec (on the stack) for the simpler forms) 5) Both blocking and non-blocking tests for I/O completion. (our test actually has a timeout value). 6) The ability to pass transport addresses around the machine without the system being involved. 7) A standard name server service to associate textual names with transport addresses. If people are really interested then I can probably send the man pages. (Though this is not the main point of this note). HOWEVER (and this is one of the points) although this system had clean, specified semantics and a fast implementation (on our hardware), it hasn't helped to sell machines, and we have now produced an NX style interface as an alternative. I think that the problem with popularising such an interface among users is that it appears to do less for them than a model with implicit buffering, and is therefore harder to start to use. The fact that it allows them greater control and can ultimately produce higher performance is not their immediate concern and does not therefore count for much. Many FORTRAN programmers in particular to do not wish to be concerned with "system programming issues" like buffer management. I would certainly like MPI to be able to support such an interface (so that we can achieve the higher performance it offers, and also make MPI applicable in non-scientific application areas). (Though those who were present in Dallas will have noted that I wasn't trying to push such an approach there, mainly because I think it's a lost cause. NX style is what we have to live with...) (Second point coming up...) However it crosses my mind that there may be some potential for embedding such an interface within the MPI model. (This is a vague thought, but I'm throwing it out so others can think about it as well). Observe that 1) Marc's persistent communication descriptors seem to remove the tag matching requirement (though they're still rather vague !) 2) One way of viewing contexts would be to implement a context as a particular queue of messages (or in other words an entirely separate communication end point) within a process. (The implications of this for contexts would be a) contexts must be declared before use b) their number may be limited c) they should be freed after use ) The combination of the two things could then come somewhere near to what Chuck is asking for... -- Jim James Cownie Meiko Limited Meiko Inc. 650 Aztec West Reservoir Place Bristol BS12 4SD 1601 Trapelo Road England Waltham MA 02154 Phone : +44 454 616171 FAX : +44 454 618188 E-Mail: jim@meiko.co.uk or jim@meiko.com From owner-mpi-pt2pt@CS.UTK.EDU Tue Jan 19 13:04:45 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA21335; Tue, 19 Jan 93 13:04:45 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA25741; Tue, 19 Jan 93 13:04:21 -0500 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Tue, 19 Jan 1993 13:04:20 EST Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from fslg8.fsl.noaa.gov by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA25733; Tue, 19 Jan 93 13:04:18 -0500 Received: by fslg8.fsl.noaa.gov (5.57/Ultrix3.0-C) id AA26371; Tue, 19 Jan 93 18:04:14 GMT Received: by macaw.fsl.noaa.gov (4.1/SMI-4.1) id AA19328; Tue, 19 Jan 93 11:03:13 MST Date: Tue, 19 Jan 93 11:03:13 MST From: hender@macaw.fsl.noaa.gov (Tom Henderson) Message-Id: <9301191803.AA19328@macaw.fsl.noaa.gov> To: mpi-pt2pt@cs.utk.edu Subject: Re: "Message Capsule" Proposal Tony Skjellum pointed out that Zipcode has been successfully using "invoices" that work in the same way as "capsules" for several years. Tony's comments were not sent to the entire mailing list and are repeated below, with his permission: > Please read about our message packing in the following uuencoded, > compressed postscript file. We already have created this technology. > (we've omitted the postscript file) > > We are able to deal effectively with fortran packing and unpacking > of messages, and 'pack+send' and 'unpack+receive' are inherently > more optimizable on some systems. > > - Tony Chuck Simmons points out that: > ... > Standards bodies work best when they standardize existing work, and work > worst when they attempt to design. > ... > > Cheers, Chuck Simmons We think that the success of "invoices" in Zipcode supports the argument that the "Message Capsule" proposal is an attempt to standardize existing work. (BTW, we aren't really that satisfied with the term "capsule". Another term may be better.) Tom Henderson Leslie Hart hender@fsl.noaa.gov hart@fsl.noaa.gov From owner-mpi-pt2pt@CS.UTK.EDU Tue Jan 19 13:13:19 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA21564; Tue, 19 Jan 93 13:13:19 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA26145; Tue, 19 Jan 93 13:12:59 -0500 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Tue, 19 Jan 1993 13:12:57 EST Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from fslg8.fsl.noaa.gov by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA26131; Tue, 19 Jan 93 13:12:55 -0500 Received: by fslg8.fsl.noaa.gov (5.57/Ultrix3.0-C) id AA26408; Tue, 19 Jan 93 18:12:52 GMT Received: by macaw.fsl.noaa.gov (4.1/SMI-4.1) id AA19336; Tue, 19 Jan 93 11:11:50 MST Date: Tue, 19 Jan 93 11:11:50 MST From: hender@macaw.fsl.noaa.gov (Tom Henderson) Message-Id: <9301191811.AA19336@macaw.fsl.noaa.gov> To: mpi-pt2pt@cs.utk.edu Subject: Re: "Message Capsule" Proposal The following is excerpts from an email discussion we have been having with Tony Skjellum regarding message "capsules" or "invoices". We have forwarded it with Tony's permission. Our comments are marked with ">>" and Tony's are marked with ">". Tom Henderson Leslie Hart hender@fsl.noaa.gov hart@fsl.noaa.gov > > Please pass along my comments. Invoices are important, in that they > permit additional possible optimizations. The more the user says > "what" and not "how," to accomplish transfer, the more opportunity for > faster performance. Invoices make Fortran-based message passing much > nicer, as well as providing more sophisticated packing/unpacking for C > people. > >> Several people (including us) were uneasy with the "hidden state" associated >> with stack-based context management. Any particular reason for using stacks >> in Zipcode? > Stacks are a detail of conext handling. They are not essential. > - Tony From owner-mpi-pt2pt@CS.UTK.EDU Tue Jan 26 18:04:14 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA23239; Tue, 26 Jan 93 18:04:14 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA26207; Tue, 26 Jan 93 18:02:56 -0500 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Tue, 26 Jan 1993 18:02:55 EST Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from timbuk.cray.com by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA26199; Tue, 26 Jan 93 18:02:53 -0500 Received: from teak18.cray.com by cray.com (4.1/CRI-MX 2.9) id AA16157; Tue, 26 Jan 93 17:02:49 CST Received: by teak18.cray.com id AA29628; 4.1/CRI-5.6; Tue, 26 Jan 93 17:02:48 CST From: par@teak.cray.com (Peter Rigsbee) Message-Id: <9301262302.AA29628@teak18.cray.com> Subject: MPI task identifiers To: mpi-pt2pt@cs.utk.edu Date: Tue, 26 Jan 93 17:02:44 CST X-Mailer: ELM [version 2.3 PL11b-CRI] At the meeting in Dallas, I asked a question about how tasks were identified, referring (for example) to the argument that one would pass to one of the variants of send. I was surprised to hear that the October draft specified this, and that it was defined as sequential integers starting at 0. I've scanned through the document since then, and didn't notice where this was stated. (The only indication I saw to such identifiers was an example included as "Table 1" on page 9, which listed "PID"s of 0-7.) Each place I'd expect to see a definition, says only "the ID number of the process..." and leaves it as that. But I'm willing to accept that it is there and that I just missed it (or the authors forgot to print it ;-). In any case, I'd like to propose that the final spec should *not* define the internals of task identifiers. Please consider this as such a proposal. I'd appreciate hearing any comments. Assuming the October spec does describe what a regular task identifier is, I'd like someone to point out where and to hear reasons why it is important to dictate a 0..n ranges for task identifiers. Thanks for your time. - Peter Rigsbee par@cray.com PROPOSAL -- OPAQUE TASK IDENTIFIERS 1. Proposal TASK IDENTIFIER All tasks in MPI are identified by unique integer identifiers that are assigned by the system upon which MPI is implemented. The MPI spec does not define the contents of these identifiers and users writing portable applications should not make any decisions about nor perform any operations based on their contents. 2. Discussion a. The major benefit of a 0..n range of identifiers is that users on many systems (most notably those supporting one process per CPU) can write efficient applications without having to deal with groups or virtual topologies. If the relationship between process identifier and process location is well-defined for a particular system, codes can operate directly on identifiers (send to "myid()+1", for example, and have the data sent to a nearest neighbor CPU). b. The drawback is that the above advantage leads programmers away from writing efficient, portable code. A code written using the convenient method above may work great on a few systems, acceptably on another set of systems, and horribly on a third set. It should get the correct answers in each case, but, for most people writing MPI codes, horrible performance would be considered a bug. (This of course assumes that the MPI specs for groups and virtual topologies will allow for efficient, portable applications. If not, we need to fix those specs, and not simply introduce a faster alternative.) c. For an MPP system with one process per CPU, a 0..n assignment is easy and natural. For almost any other system, it will be more complex. For a network implementation, it may require the use of a "process-identifier server" or some sort of globally shared "next identifier" variable. These can be very costly to implement. It also raises ugly questions about where and how to map these identifiers to the information actually locating the task. d. In contrast, the proposal allows an implementation to use the most appropriate and efficient method. It may simply be a Unix PID. It could be a range of 0..n integers. Or it could be a complex encoding of host/PID information that would allow a network implementation to get better performance. It's up to the implementation. e. The MPI spec does not address the fact that some implementations may be provided on top of systems allowing dynamic creation and termination of tasks. A 0..n requirement assumes a static system. What happens when task goes away? There is no longer a sequential list from 0..n, so does this implemenation still adhere to the MPI spec? (If so, one could argue that the 0..n spec is then just a red herring, since one could assign any numbers at all, and claim simply that all missing numbers represented tasks that never started.) With the proposal, the answer is simple -- that identifier no longer represents a task, and the implementation still adheres to the spec. f. The above proposal does not prohibit implementations from assigning 0..n identifiers, or for users of such implementations relying on this assignment. But it makes it clear that such reliance is non-standard and non-portable. 3. Summary The above proposal has a number of benefits, particularly when compared to the 0..n alternative: - allows implementations to use the most appropriate and/or efficient method for assigning task identifiers (including 0..n) - encourages portable programming that does not rely on operations on task identifiers - no confusing semantics for MPI implementations built on top of dynamic systems From owner-mpi-pt2pt@CS.UTK.EDU Tue Jan 26 19:25:11 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA23808; Tue, 26 Jan 93 19:25:11 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA29304; Tue, 26 Jan 93 19:24:15 -0500 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Tue, 26 Jan 1993 19:24:14 EST Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from Aurora.CS.MsState.Edu by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA29296; Tue, 26 Jan 93 19:24:13 -0500 Received: by Aurora.CS.MsState.Edu (4.1/6.0s-FWP); id AA05866; Tue, 26 Jan 93 18:24:11 CST Date: Tue, 26 Jan 93 18:24:11 CST From: Tony Skjellum Message-Id: <9301270024.AA05866@Aurora.CS.MsState.Edu> To: mpi-pt2pt@cs.utk.edu, par@teak.cray.com Subject: Re: MPI task identifiers Concerning Peter's proposal... It may have been a mistake to simplify the naming of processes to a single integer when MPI was defined. Commonly, systems like RK, and so on had used pairs of integers on multicomputers and networks before that; the two numbers did have semantic content: the first number, a node number, connoted the location of the process. The second, a PID, could provide an implementation-dependent way to name a process on a processor. Hence, at the low level, specification of WHERE processes ran was permitted, but it was not necessary to restrict how they were named (beyond it being an integer quantity) on a processor. At a low level, it is convenient to be able to control process placement. When processes are created, the user could specify the node and PID on some systems, with either or both picked by the system on some implementations. Such composite information could then be used as part of message passing. nCUBE, for instance, specifies the PID's but does not restrict the locations (nodes are selected by user). The Mosaic system will generally place processes at will, and a message-passing interface for it will have neither option. For scalable spawning of large numbers of identical processes, some mechanism of this form is needed (user specifies collection of process locations, gets back identifier describing these as a group; eg, a common PID). In early systems, and arguably still, it is important to provide low-level control over process placement. Arguably, this need not be done in the message passing interface, but it must be done portably somewhere in the system. Otherwise, it will be difficult to provide guarantees of performance behavior (ie, force one process per processor, if desired; clump processes together to get overlapping capability when desired, etc). Our Zipcode system uses {node,pid} in addressee lists, but these are not used by the programmer directly, in most message classes (eg, 3D logical process grids) but are hidden with further mappings to application-relevant names for processes. In fact, we would like to generalize these addressee lists to include other information besides this naming, but that remains a future-release feature. It does not matter so much to us what convention names processes, but it is interesting to have access to semantic information about processes named by the system (eg, accessors to location, etc). Such information would be used when providing services, like Zipcode, that are higher level. The user might not want them at all for direct use. Forbidding a user code or application layer like Zipcode to take advantage of architecture (albeit transparently to the user code) is a disadvantage. Typically, process placement remains an aspect of architecture we need to control. Just some thoughts... - Tony Skjellum From owner-mpi-pt2pt@CS.UTK.EDU Wed Jan 27 08:37:57 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA02588; Wed, 27 Jan 93 08:37:57 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA01030; Wed, 27 Jan 93 08:37:18 -0500 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Wed, 27 Jan 1993 08:37:17 EST Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from rios2.EPM.ORNL.GOV by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA01022; Wed, 27 Jan 93 08:37:16 -0500 Received: by rios2.epm.ornl.gov (AIX 3.2/UCB 5.64/4.03) id AA17101; Wed, 27 Jan 1993 08:37:15 -0500 Date: Wed, 27 Jan 1993 08:37:15 -0500 From: walker@rios2.epm.ornl.gov (David Walker) Message-Id: <9301271337.AA17101@rios2.epm.ornl.gov> To: mpi-pt2pt@cs.utk.edu Subject: Task Indentifers I assume that "task" in Peter's proposal are the same as "processes" elsewhere in the MPI discussion. I think there is some confusion over the terms "PID" and "rank". If PIDS do not convey any information to the application then as Peter says: "users writing portable applications should not make any decisions about nor perform any operations based on [task identifiers]." In fact there's almost no point in having task identifiers accessible to an application. Instead applications typically use what was referred to at the Dallas meeting as the "rank", which is a one-to-one mapping of PIDs to the integers 0,1,2,...n-1. The user needs ranks so s/he can say things like, if ( myID() .EQ. 0 ) output(data) PID (or task ID if you like) and rank are synonymous in MPI1 since we could see no reason for distinguishing them. Note that "rank" is essentially an arbitrary enumeration of processes, and contains no information about relative placement of processes on physical processors. A ranking is the same as a 1-D topology. MPI does not currently address how a topology is mapped to physical processors, in fact the whole topic of process creation has so far been avoided. I guess my main question with Peter's proposal is : "If an application shouldn't make any decisions about nor perform any operations based on task IDs, then why does an application even need to know about task IDs?" From owner-mpi-pt2pt@CS.UTK.EDU Wed Jan 27 09:00:06 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA03235; Wed, 27 Jan 93 09:00:06 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA01779; Wed, 27 Jan 93 08:59:44 -0500 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Wed, 27 Jan 1993 08:59:43 EST Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from marge.meiko.com by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA01771; Wed, 27 Jan 93 08:59:40 -0500 Received: from hub.meiko.co.uk by marge.meiko.com with SMTP id AA00815 (5.65c/IDA-1.4.4 for ); Wed, 27 Jan 1993 08:59:36 -0500 Received: from float.co.uk (float.meiko.co.uk) by hub.meiko.co.uk (4.1/SMI-4.1) id AA13095; Wed, 27 Jan 93 13:59:32 GMT Date: Wed, 27 Jan 93 13:59:32 GMT From: jim@meiko.co.uk (James Cownie) Message-Id: <9301271359.AA13095@hub.meiko.co.uk> Received: by float.co.uk (5.0/SMI-SVR4) id AA02254; Wed, 27 Jan 93 13:58:07 GMT To: walker@rios2.epm.ornl.gov Cc: mpi-pt2pt@cs.utk.edu In-Reply-To: David Walker's message of Wed, 27 Jan 1993 08:37:15 -0500 <9301271337.AA17101@rios2.epm.ornl.gov> Subject: Opaque Task Identifers Content-Length: 1945 I tend to agree with David here. It is clear that the underlying system level identification of a process can potentially be arbitrarily complex, and will almost certainly require at least a (node,process within node) tuple. However, this is not the way the user wants to think about her program. She wants to think about a set of processors with an obvious mapping onto the positive integers. (Fortran programmers would probably prefer that to read "strictly positive integers" ...). At the user level we shouldn't be exposing the internals of the system's process mapping scheme, since it will likely be different on each machine. (On a net doubtless there will be whole ip addresses kicking around, and so on). I would suggest that we let the user think about a simple enumeration of processors, but stipulate that this is a random enumeration (in other words there is no guarantee whatever that an N to N+1 communication is faster than an N to N+x communication). The system can then use a simple table lookup to translate from the user's process number into the system address. Most of the time this will be necessary anyway (on all systems which need more than this simple numbering scheme), and in any case it's only a single (unsigned) compare followed by one store access. If the user needs to have guarantees of "nearness", then she should use the virtual topologies mechanism to obtain a good process enumeration. If we were feeling mean, then we'd add an implementation recommendation that the intial mapping should be random, so that people get an incentive to use the virtual topology mechanisms even to implement a straight line of processes. -- Jim James Cownie Meiko Limited Meiko Inc. 650 Aztec West Reservoir Place Bristol BS12 4SD 1601 Trapelo Road England Waltham MA 02154 Phone : +44 454 616171 +1 617 890 7676 FAX : +44 454 618188 +1 617 890 5042 E-Mail: jim@meiko.co.uk or jim@meiko.com From owner-mpi-pt2pt@CS.UTK.EDU Wed Jan 27 10:29:22 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA05985; Wed, 27 Jan 93 10:29:22 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA05149; Wed, 27 Jan 93 10:28:53 -0500 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Wed, 27 Jan 1993 10:28:51 EST Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from watson.ibm.com by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA05141; Wed, 27 Jan 93 10:28:50 -0500 Message-Id: <9301271528.AA05141@CS.UTK.EDU> Received: from YKTVMV by watson.ibm.com (IBM VM SMTP V2R2) with BSMTP id 7863; Wed, 27 Jan 93 10:28:42 EST Date: Wed, 27 Jan 93 09:50:23 EST From: "Shlomo Kipnis ((914) 945-1281)" To: mpi-pt2pt@cs.utk.edu Subject: Process enumeration The main goal of the MPI standard is the portability of applications between different machines. Linear ordering of process (task) ids in the range 0..n is the simplest and most portable numbering scheme I can think of. As David and Jim state, this numbering scheme is used by many applications, and most machines provide a virtual domain of processes in the range 0..n to be used. Any implementation of MPI, obviously, need to translate this numbering scheme to the physical process/task/thread ids used in the underlying system. However, a portable user's application cannot rely on such a physical numbering scheme. The performance of an application may be very different on different machines, but at least an application that uses MPI is guaranteed to run on different machines (unlike the state of the art today). An interesting feature of the "process groups" idea is that pids in a process group can always be addressed in the range of 0..n using the ranks of processes within the group. By using process groups, one can avoid the issue of noncontiguous pids range which may be caused by particular partitions of the machine, processes leaving or joining an application, running several application codes concurrently under the same job, etc... As currently defined, process groups do not address the issue of an application's topology. However, topologies can be defined on top of process groups. (I hope the topology subcommittee discusses these issues.) To summarize, I think that process ids in the range 0..n are the right approach for ensuring portability of applications. This scheme also addresses the virtual organization of processes in a process group. The virtual-to-physical translation of pids is done in many systems and is typically very fast. Shlomo Kipnis IBM T.J Watson Research Center Yorktown Heights, NY 10598 (914) 945-1281 kipnis@watson.ibm.com From owner-mpi-pt2pt@CS.UTK.EDU Thu Jan 28 09:12:32 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA08710; Thu, 28 Jan 93 09:12:32 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA02034; Thu, 28 Jan 93 09:11:44 -0500 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Thu, 28 Jan 1993 09:11:43 EST Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from msr.EPM.ORNL.GOV by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA02026; Thu, 28 Jan 93 09:11:42 -0500 Received: by msr.EPM.ORNL.GOV (5.67/1.34) id AA03474; Thu, 28 Jan 93 09:11:39 -0500 Date: Thu, 28 Jan 93 09:11:39 -0500 From: geist@msr.EPM.ORNL.GOV (Al Geist) Message-Id: <9301281411.AA03474@msr.EPM.ORNL.GOV> To: mpi-pt2pt@cs.utk.edu Subject: My two cents worth on process numbering. >To summarize, I think that process ids in the range 0..n are the right >approach for ensuring portability of applications. With the ability to have multiple groups each with processes numbered 0...p then no process can be uniquely identified by a single integer [0,1,...n]. The above process naming scheme will require some group context information down in the point to point routines. It can be explicit as in a (group_id, rank) pair which I prefer because it makes codes more readable and easier to debug, or it can be implicit as in the original MPI draft. Peter Rigsbee is correct in noting that on large MPP systems looking up this mapping of 0-p to real process location and id can be expensive in both time and memory. If the high priests want to be efficient and go fast then Peter's suggestion of having a more sophisticated task naming should be considered seriously. Al Geist From owner-mpi-pt2pt@CS.UTK.EDU Thu Jan 28 09:18:28 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA08787; Thu, 28 Jan 93 09:18:28 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA02247; Thu, 28 Jan 93 09:18:02 -0500 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Thu, 28 Jan 1993 09:18:01 EST Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from gmdzi.gmd.de by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA02239; Thu, 28 Jan 93 09:17:56 -0500 Received: from f1neuman.gmd.de (f1neuman) by gmdzi.gmd.de with SMTP id AA24455 (5.65c/IDA-1.4.4 for ); Thu, 28 Jan 1993 15:16:56 +0100 Received: by f1neuman.gmd.de id AA15219; Thu, 28 Jan 1993 15:16:01 +0100 Date: Thu, 28 Jan 1993 15:16:01 +0100 From: Rolf.Hempel@gmd.de Message-Id: <9301281416.AA15219@f1neuman.gmd.de> To: mpi-pt2pt@cs.utk.edu Subject: MPI task identifiers Cc: gmap10@f1neuman.gmd.de During the last two days there was an interesting discussion about the naming of processes in MPI. I just want to throw some thoughts into the discussion. If we number the processes in a group from 0 to n-1, I see two benefits: 1. The applications programmer immediately knows how to address the other processes in the same group, instead of first having to look up some strange id numbers. Sure, the translation to the underlying (h/w) or network addresses has to be done somewhere, but it's not necessarily the applications programmer who wants to do this. 2. The 0,...,n-1 ranking of processes provides a natural order for collective communications, like gather. Without nicely ordered id numbers, almost every reasonable MPI program would have to use the virtual topologies. If you look at the proposal of the PTOP subcommittee, you will find no assumption about the definition of the process identifiers in a group. All ordering and addressing of processes is provided by the topologies. So, as a proponent of virtual topologies I could be happy with that solution. At the Dallas meeting somebody already suggested that there be always a topology associated with a process group. At group creation this would be just a linear ordering. Since topologies can be overwritten, this default could be changed by the user at any time. Note that this would not mean that the pid numbers are arranged as 0,..,n-1. A process would use the topology look-up functions to get the pids of its neighbors, whereas the pid numbers could have any value (as Peter Ringsbee proposes). I think we have to choose between: a. We force programmers to use the topology functions in virtually all applications. The pid numbers can then have any values. The default topology of a group is a linear ordering. The mapping of processes to processors causes some (usually small) overhead at group creation. b. The pid numbers are numbered as 0,..,n-1. This does not mean the processes are mapped in a kind of nearest-neighbor fashion to the processors. It is rather a (random) enumeration of the processes in a group, in order to get benefits 1. and 2. (see above). If one wants to have a good mapping, one should use the virtual topology functions (which I recommend, of course). On the other hand, this introduces another level of address indirection. I tend towards alternative b., although there are some tricky problems. For example: when a topology is assigned to a group, does this then change the pid numbers? If not, then the buffer structure in collective operations will not be aligned with the topology. On the other hand, changing the pid number of a process looks ugly to me. Perhaps it helps if we avoid the word "pid" alltogether, and only refer to the "rank" of a process, as David Walker suggests. I suggest that at the next Dallas meeting we have a session in which the subcommittees for collective communication and process topologies meet together. To me this seems to be the ideal forum where we can try to solve the problem. Rolf Hempel From owner-mpi-pt2pt@CS.UTK.EDU Thu Jan 28 09:57:56 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA09245; Thu, 28 Jan 93 09:57:56 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA04162; Thu, 28 Jan 93 09:57:00 -0500 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Thu, 28 Jan 1993 09:56:59 EST Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from marge.meiko.com by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA04154; Thu, 28 Jan 93 09:56:56 -0500 Received: from hub.meiko.co.uk by marge.meiko.com with SMTP id AA11324 (5.65c/IDA-1.4.4 for ); Thu, 28 Jan 1993 09:56:52 -0500 Received: from float.co.uk (float.meiko.co.uk) by hub.meiko.co.uk (4.1/SMI-4.1) id AA25589; Thu, 28 Jan 93 14:56:47 GMT Date: Thu, 28 Jan 93 14:56:47 GMT From: jim@meiko.co.uk (James Cownie) Message-Id: <9301281456.AA25589@hub.meiko.co.uk> Received: by float.co.uk (5.0/SMI-SVR4) id AA02620; Thu, 28 Jan 93 14:55:18 GMT To: geist@msr.EPM.ORNL.GOV Cc: mpi-pt2pt@cs.utk.edu In-Reply-To: Al Geist's message of Thu, 28 Jan 93 09:11:39 -0500 <9301281411.AA03474@msr.EPM.ORNL.GOV> Subject: Re Al's two cents worth on process numbering. Content-Length: 2327 (Come on Al give us the whole $1 worth !) Al asserts that > Peter Rigsbee is correct in noting that on large MPP systems looking > up this mapping of 0-p to real process location and id can be > expensive in both time and memory. I disagree that this is necessarily expensive, for the following reasons :- 1) At some level the system has to know the lowest level address, otherwise it cannot send the data. (I know this could be a route, rather than an absolute address, but it comes to the same thing) So it's a cost you have to pay somewhere. 2) In the worst case the cost is constant in time and linear in number of nodes in memory. (Just index into an array of low level addresses). How bad is this really ? Suppose you have a 16K node machine with 128 bit low level addresses (4 32bit words) this occupies 256K bytes, which doesn't seem so bad. 3) You can trade off the store requirement against a larger time cost. Rather than holding all of the destination addresses you implement an address cache (using standard symbol table technology), and fill in the addresses as you use them. (Of course you need a distributed scalable lookup to find them the first time). This then requires store which depends on the working set of targets for communication, and whatever CPU time is taken in the lookup. (Should be logarithmic in number of processors if you use a tree). The issue I'd like to clarify here is what is actually being asked for by Peter ? Is he asking that the user deal with genuine low level addresses (which are potentially of arbitrary size, and certainly different on different vendors machines) ? or Is he asking that users deal with integer identifiers, but these are opaque in the sense that modifying then makes no sense ? The former seems extremely unattractive (especially in Fortran !), and if it's the latter, then you still have to do a translation somewhere to get to the real low level address, so why not let the user have what they really wanted (i.e. 0..N) ? -- Jim James Cownie Meiko Limited Meiko Inc. 650 Aztec West Reservoir Place Bristol BS12 4SD 1601 Trapelo Road England Waltham MA 02154 Phone : +44 454 616171 +1 617 890 7676 FAX : +44 454 618188 +1 617 890 5042 E-Mail: jim@meiko.co.uk or jim@meiko.com From owner-mpi-pt2pt@CS.UTK.EDU Fri Feb 5 00:33:35 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA29228; Fri, 5 Feb 93 00:33:35 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA27483; Fri, 5 Feb 93 00:31:56 -0500 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Fri, 5 Feb 1993 00:31:54 EST Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from relay2.UU.NET by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA27473; Fri, 5 Feb 93 00:31:52 -0500 Received: from uunet.uu.net (via LOCALHOST.UU.NET) by relay2.UU.NET with SMTP (5.61/UUNET-internet-primary) id AA09343; Fri, 5 Feb 93 00:31:59 -0500 Received: from kailand.UUCP by uunet.uu.net with UUCP/RMAIL (queueing-rmail) id 002919.13027; Fri, 5 Feb 1993 00:29:19 EST Received: from brisk.kai.com (brisk) by kailand.kai.com via SMTP (5.65d-93012701) id AA09357; Thu, 4 Feb 1993 19:11:45 -0600 Received: by brisk.kai.com (920330.SGI-92101201) id AA05200; Thu, 4 Feb 93 19:11:44 -0600 Date: Thu, 4 Feb 93 19:11:44 -0600 Message-Id: <9302050111.AA05200@brisk.kai.com> To: mpi-formal@cs.utk.edu, mpi-pt2pt@cs.utk.edu Cc: Subject: Composition. From: Steven Ericsson Zenith Sender: zenith@kai.com Organization: Kuck and Associates, Inc. 1906 Fox Drive, Champaign IL USA 61820-7334, voice 217-356-2288, fax 217-356-5199 What is the meaning of these sequential compositions? 1) two non-blocking sends to the same destination? 2) a non-blocking send and a blocking send to the same destination? 3) a non-blocking send and a broadcast? 4) two non-blocking receives? 5) a non-blocking receive and a blocking receive? and so on ... I suspect different machine vendors may have different views. Steven ps. mpi-formal members are refered to mpi-pt2pt. From owner-mpi-pt2pt@CS.UTK.EDU Fri Feb 5 00:33:39 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA29230; Fri, 5 Feb 93 00:33:39 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA27500; Fri, 5 Feb 93 00:32:03 -0500 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Fri, 5 Feb 1993 00:32:01 EST Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from relay2.UU.NET by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA27492; Fri, 5 Feb 93 00:31:59 -0500 Received: from uunet.uu.net (via LOCALHOST.UU.NET) by relay2.UU.NET with SMTP (5.61/UUNET-internet-primary) id AB09405; Fri, 5 Feb 93 00:32:06 -0500 Received: from kailand.UUCP by uunet.uu.net with UUCP/RMAIL (queueing-rmail) id 002916.13009; Fri, 5 Feb 1993 00:29:16 EST Received: from brisk.kai.com (brisk) by kailand.kai.com via SMTP (5.65d-93012701) id AA09184; Thu, 4 Feb 1993 18:56:01 -0600 Received: by brisk.kai.com (920330.SGI-92101201) id AA05164; Thu, 4 Feb 93 18:56:00 -0600 Date: Thu, 4 Feb 93 18:56:00 -0600 Message-Id: <9302050056.AA05164@brisk.kai.com> To: mpi-formal@cs.utk.edu, mpi-pt2pt@cs.utk.edu Cc: Subject: Concerns. From: Steven Ericsson Zenith Sender: zenith@kai.com Organization: Kuck and Associates, Inc. 1906 Fox Drive, Champaign IL USA 61820-7334, voice 217-356-2288, fax 217-356-5199 I have to agree with Steve Otto and confess more than a little anxiety at this end. It seems to me that we are much too concerned with being all things to all men and I am absolutely convinced that a mix and match philosophy is a recipe for disaster: not that it can't be implemented or specified but purely from the usability point of view. We should be able to provide more wizardly and wise power by providing less not more! We would do better to ensure that the issues we address are those that we all currently understand well or that we have time to defer to assure ourselves that new forms or compositions are at least well concieved. The latter is impossible in the six months we have allotted ourselves. If we really want people to use MPI it has to be straight forward and at least no more complex than existing systems. If we can't decide on a *concise* set of wisely restricted forms for send and recieve, if you really believe we have to provide all forms, then I suggest this effort may be premature. Steven From owner-mpi-pt2pt@CS.UTK.EDU Fri Feb 5 05:48:31 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA13673; Fri, 5 Feb 93 05:48:31 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA18966; Fri, 5 Feb 93 05:44:22 -0500 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Fri, 5 Feb 1993 05:44:21 EST Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from relay2.UU.NET by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA18958; Fri, 5 Feb 93 05:44:20 -0500 Received: from uunet.uu.net (via LOCALHOST.UU.NET) by relay2.UU.NET with SMTP (5.61/UUNET-internet-primary) id AA24268; Fri, 5 Feb 93 05:44:19 -0500 Received: from kailand.UUCP by uunet.uu.net with UUCP/RMAIL (queueing-rmail) id 054326.7477; Fri, 5 Feb 1993 05:43:26 EST Received: from brisk.kai.com (brisk) by kailand.kai.com via SMTP (5.65d-93012701) id AA16580; Thu, 4 Feb 1993 23:45:25 -0600 Received: by brisk.kai.com (920330.SGI-92101201) id AA06410; Thu, 4 Feb 93 23:45:23 -0600 Date: Thu, 4 Feb 93 23:45:23 -0600 Message-Id: <9302050545.AA06410@brisk.kai.com> To: mpi-formal@cs.utk.edu, mpi-pt2pt@cs.utk.edu Cc: Subject: Be brave. Be sure. From: Steven Ericsson Zenith Sender: zenith@kai.com Organization: Kuck and Associates, Inc. 1906 Fox Drive, Champaign IL USA 61820-7334, voice 217-356-2288, fax 217-356-5199 It will probably come as no surprise that I, for one, have no fear of being politically incorrect in these matters ;-) I am not afraid to say that I think at this stage it would be a very good idea to define only process composition and blocking send/recv. The rest comes - more or less - as a construction of the other two. Process ids are simply a many to one channel descriptor - implementable by a single process, non-synchronized communication is simply a process creation, broadcast is a composition of N sends with N receives, and such an interface can be implemented by all the available technologies. In addition this simplification would greatly aid the task of the poor programmer. Where is the innovation in this forum? It seems that we are pandering to the lowest common denominator. Has noone learnt anything in the past 7/8 years? Otherwise we should simply take an existing system (PVM for example, though there are many well designed MP systems) and adopt it as the accepted standard, work from that point and be done with it. The interface efforts of the Unix community (e.g., DCE and sockets) may well out run this one. Steven From owner-mpi-pt2pt@CS.UTK.EDU Fri Feb 5 06:05:42 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA17447; Fri, 5 Feb 93 06:05:42 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA20412; Fri, 5 Feb 93 06:04:52 -0500 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Fri, 5 Feb 1993 06:04:51 EST Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from marge.meiko.com by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA20403; Fri, 5 Feb 93 06:04:45 -0500 Received: from hub.meiko.co.uk by marge.meiko.com with SMTP id AA27476 (5.65c/IDA-1.4.4); Fri, 5 Feb 1993 06:04:40 -0500 Received: from float.co.uk (float.meiko.co.uk) by hub.meiko.co.uk (4.1/SMI-4.1) id AA16231; Fri, 5 Feb 93 11:04:37 GMT Date: Fri, 5 Feb 93 11:04:36 GMT From: jim@meiko.co.uk (Paul Kelly) Message-Id: <9302051104.AA16231@hub.meiko.co.uk> Received: by float.co.uk (5.0/SMI-SVR4) id AA06680; Fri, 5 Feb 93 11:02:53 GMT To: zenith@kai.com Cc: mpi-formal@cs.utk.edu, mpi-pt2pt@cs.utk.edu In-Reply-To: Steven Ericsson Zenith's message of Thu, 4 Feb 93 23:45:23 -0600 <9302050545.AA06410@brisk.kai.com> Subject: Be brave. Be sure. Content-Length: 896 Steve, > I am not afraid to say that I think at this stage it would be a very > good idea to define only process composition and blocking send/recv. > The rest comes - more or less - as a construction of the other two. > Process ids are simply a many to one channel descriptor - > implementable by a single process, non-synchronized communication is > simply a process creation, broadcast is a composition of N sends with > N receives, and such an interface can be implemented by all the > available technologies. You just described occam. One thing I have learnt from the last 10 years is that that's NOT ENOUGH. -- Jim James Cownie Meiko Limited Meiko Inc. 650 Aztec West Reservoir Place Bristol BS12 4SD 1601 Trapelo Road England Waltham MA 02154 Phone : +44 454 616171 +1 617 890 7676 FAX : +44 454 618188 +1 617 890 5042 E-Mail: jim@meiko.co.uk or jim@meiko.com From owner-mpi-pt2pt@CS.UTK.EDU Fri Feb 5 08:24:12 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA02543; Fri, 5 Feb 93 08:24:12 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA27610; Fri, 5 Feb 93 08:23:34 -0500 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Fri, 5 Feb 1993 08:23:33 EST Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from sol.cs.wmich.edu by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA27602; Fri, 5 Feb 93 08:23:31 -0500 Received: from id.wmich.edu by cs.wmich.edu (4.1/SMI-4.1) id AA19627; Fri, 5 Feb 93 08:17:03 EST Date: Fri, 5 Feb 93 08:17:03 EST From: john@cs.wmich.edu (John Kapenga) Message-Id: <9302051317.AA19627@cs.wmich.edu> To: mpi-pt2pt@CS.UTK.EDU hi; A good offline note to me from > From: Alan Mainwaring together with some other recent posts, indicates that we are using at least two valid but very different meanings for the term "overlappping communication and computation". -------- 1) Overlapping IO and computation means transfers from the communication processor to and from the node's computation processor memory are carried out by the node's memory system system while a computation is actively running on the node. 2) Overlapping IO and computation means receives and sends requests in a process may be pending while a computation is actively running in a process. -------- Note 2) does not require 1) , some ways 2) could (has) been provided: 2) could be cone by 1) 2) could be done by the OS while the unknowing computation process is interupted and placed in a wait state by the OS when it wants to prefrom a transfer. 2) could be done when the computation process makes any system call. -------- I believe MPI must have 2), that is non-blocking sends and receives. I also think it would pass any straw vote by the widest of margins, I do not have any real feeling about how important 1) currently is. I would expect 1) could become an issue if communications become fast enough to really support distributed virtual memory (that is, commmunication speed approaches computation speed). Then much finer grained parallelism can be applied. It would seem that most interfaces that support direct transfers into memory, rather than a buffer (iovec, stride etc.), could logically also well support 2), if the hardware provides it. It would be best if the interface specifed could use 2) if available, without extra complication. cheers - john From owner-mpi-pt2pt@CS.UTK.EDU Fri Feb 5 13:13:33 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA09267; Fri, 5 Feb 93 13:13:33 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA16600; Fri, 5 Feb 93 13:09:51 -0500 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Fri, 5 Feb 1993 13:09:50 EST Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from relay2.UU.NET by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA16588; Fri, 5 Feb 93 13:09:48 -0500 Received: from uunet.uu.net (via LOCALHOST.UU.NET) by relay2.UU.NET with SMTP (5.61/UUNET-internet-primary) id AA25537; Fri, 5 Feb 93 13:09:50 -0500 Received: from kailand.UUCP by uunet.uu.net with UUCP/RMAIL (queueing-rmail) id 130847.19757; Fri, 5 Feb 1993 13:08:47 EST Received: from brisk.kai.com (brisk) by kailand.kai.com via SMTP (5.65d-93012701) id AA04173; Fri, 5 Feb 1993 10:16:24 -0600 Received: by brisk.kai.com (920330.SGI-92101201) id AA07622; Fri, 5 Feb 93 10:16:22 -0600 Date: Fri, 5 Feb 93 10:16:22 -0600 Message-Id: <9302051616.AA07622@brisk.kai.com> To: jim@meiko.co.uk Cc: mpi-formal@cs.utk.edu, mpi-pt2pt@cs.utk.edu In-Reply-To: Paul Kelly's message of Fri, 5 Feb 93 13:20:43 GMT <9302051320.AA16757@hub.meiko.co.uk> Subject: Be brave. Be sure. From: Steven Ericsson Zenith Sender: zenith@kai.com Organization: Kuck and Associates, Inc. 1906 Fox Drive, Champaign IL USA 61820-7334, voice 217-356-2288, fax 217-356-5199 Thanks Jim (or is it Paul) for the name correction (via private mail)... I know what I'm suggesting - and yes, though I hate to admit it, it is the Occam model more or less. Though you shouldn't allow your distaste for the language to deflect you from the qualities of it's communication model. The inadequacies of that language are many but when it comes to the communication model (CSP) it is fundamental. All I'm suggesting here is that we do not make the other mistake, that of providing so much that it is neither managable by us or our intended audience. Either we are prepared to provide something better than currently exists based on a good understanding of our collective experience with mature concepts or we should start from an existing system, with all its flaws, and work from there. This is not a forum for invention. I should qualify my comments to reiterate that in fact I am looking to this forum to provide a consistent interface for low level implementation in operating systems and code generation. I never want any of my general purpose users using message passing as a means to program machines. And if you gave me just processes and blocking send and receive that *would* be enough. As someone who has written many message passing programs (using Express and other libraries, as well as Occam and CSP), and made it my business over the years to talk to others who have, I have to tell you that the fewer well conceived choices you give me the happier I am and, in my experience, the user is. I just don't believe those people who say we have to provide every form - it is counter to all my experience with users (no doubt, there are users here who will insist on being provided with every variant ;-) - but you will have seen that manifestation before. ). One way to look for evidence to support or denounce this is to undertake an investigation. Find an array of *applications*, not systems tools etc. but "real" applications, and identify the subset of functionality used. My bet is that you will discover the subset is small, the place where non-blocking communication is used is on machines where the process model implementation is inadequate, and that code that does not fit into my simple model is not readily portable. Steven From owner-mpi-pt2pt@CS.UTK.EDU Fri Feb 5 14:07:52 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA10311; Fri, 5 Feb 93 14:07:52 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA20802; Fri, 5 Feb 93 14:07:06 -0500 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Fri, 5 Feb 1993 14:07:05 EST Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from cs.sandia.gov by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA20794; Fri, 5 Feb 93 14:07:03 -0500 Received: from panther.cs.sandia.gov by cs.sandia.gov (4.1/SMI-4.1) id AA11334; Fri, 5 Feb 93 12:06:57 MST Received: by panther.cs.sandia.gov (Smail3.1.28.1 #1) id m0nKYO9-0016ZKC; Fri, 5 Feb 93 12:06 MST Message-Id: Date: Fri, 5 Feb 93 12:06 MST From: srwheat@cs.sandia.gov (Stephen R. Wheat) To: mpi-pt2pt@cs.utk.edu Subject: Re: Be brave. Be sure. Steven: While I am impressed that you have enough influence over your users to convince them that they need not see the messaging environment themselves, I must confess that I am not so fortunate. My users are developing their codes in C, F77, and C++. The current trend here at Sandia is towards C++. Our REAL codes (REAL code is one that we make money off of, hence very important :-) are done by those that are still waiting for compilers to do everything. But, while they continue to wait, their livelyhoods depend upon their ability to get the best performance from these MP beasts as possible. Even assembly language is still a useful tool for them, case in point: tuned libraries where interfaces are defined but the library code is certainly not portable. In addition, it is my opinion that there is way too little experience with compiler generated messaging to have such a concept meet the "maturity/experience" criteria upon which we wish to base MPI. As for overlapped messaging, I dare say that EVERY code here at Sandia makes use of overlapped messaging. At the very least, the programmers want to have multiple I/O channels active simultaneously, which is at least possible on our two 1024-processor nCUBE 2 hypercubes. Furthermore, it is clear that my users are already digging into what possibilities for performance enhancements await them in our new Paragon. These are general users! Not just the wizards. Again, the "food chain" is quite clear. We do want simply understood features for the "novice" or "don't care" people. However, we must have what the experienced general users want. Furthermore, we must have the "buy-in" of the wizards. To make this more clear, I use the term wizard to refer to application types doing real, dollar/pound income-oriented work. That is, they are not of the CS lunatic fringe type, but are real-world applications people. All of this is just to state that the standard should be primarily for human use, not for compilers only. As has been voiced by others, I strongly suspect that overlapped I/O and computation is a feature that general users will (AND DO) want. Furthermore, I believe that "our collective experience with mature concepts" can tackle this problem in a very successful manner. Stephen From owner-mpi-pt2pt@CS.UTK.EDU Fri Feb 5 17:00:18 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA13516; Fri, 5 Feb 93 17:00:18 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA01026; Fri, 5 Feb 93 16:59:06 -0500 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Fri, 5 Feb 1993 16:59:05 EST Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from Aurora.CS.MsState.Edu by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA01011; Fri, 5 Feb 93 16:59:03 -0500 Received: by Aurora.CS.MsState.Edu (4.1/6.0s-FWP); id AA00846; Fri, 5 Feb 93 15:58:52 CST Date: Fri, 5 Feb 93 15:58:52 CST From: Tony Skjellum Message-Id: <9302052158.AA00846@Aurora.CS.MsState.Edu> To: mpi-pt2pt@cs.utk.edu, srwheat@cs.sandia.gov Subject: Re: Be brave. Be sure. To follow up on Stephen's comment, I do not believe that compilers would make good use of this standard. Active-message-level programming seems a much better level for standardizing compiler interfaces. - Tony From owner-mpi-pt2pt@CS.UTK.EDU Fri Feb 5 18:13:41 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA14881; Fri, 5 Feb 93 18:13:41 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA04250; Fri, 5 Feb 93 18:13:18 -0500 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Fri, 5 Feb 1993 18:13:17 EST Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from relay2.UU.NET by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA04238; Fri, 5 Feb 93 18:13:15 -0500 Received: from uunet.uu.net (via LOCALHOST.UU.NET) by relay2.UU.NET with SMTP (5.61/UUNET-internet-primary) id AA24078; Fri, 5 Feb 93 18:13:19 -0500 Received: from kailand.UUCP by uunet.uu.net with UUCP/RMAIL (queueing-rmail) id 181248.24121; Fri, 5 Feb 1993 18:12:48 EST Received: from brisk.kai.com (brisk) by kailand.kai.com via SMTP (5.65d-93012701) id AA01338; Fri, 5 Feb 1993 17:10:16 -0600 Received: by brisk.kai.com (920330.SGI-92101201) id AA08601; Fri, 5 Feb 93 17:10:15 -0600 Date: Fri, 5 Feb 93 17:10:15 -0600 Message-Id: <9302052310.AA08601@brisk.kai.com> To: mpi-formal@cs.utk.edu, mpi-pt2pt@cs.utk.edu Cc: In-Reply-To: Steven Ericsson Zenith's message of Fri, 5 Feb 93 10:16:22 EST Subject: Be brave. Be sure. From: Steven Ericsson Zenith Sender: zenith@kai.com Organization: Kuck and Associates, Inc. 1906 Fox Drive, Champaign IL USA 61820-7334, voice 217-356-2288, fax 217-356-5199 From: srwheat@cs.sandia.gov (Stephen R. Wheat) ... But, while they continue to wait, their livelyhoods depend upon their ability to get the best performance from these MP beasts as possible. Even assembly language is still a useful tool for them, case in point: tuned libraries where interfaces are defined but the library code is certainly not portable. The fact that your users are programming using message passing suggests that they are certainly also prepared to use assembler when necessary. The reason they use MP is the same reason they use assembler at times: there is no higher level way currently to achieve what they want with the performance they require on the machines they are using. ;-) Evenso, when I've asked people (physicists and so on) with some experience programming such machines this question they have confirmed that they use MP as reluctantly as they use assembler - it's just that sometimes it's unavoidable. Some of those people are at Sandia. I certainly would not want to deprive them of their lively hood and there are exceptions - of course. Some people really enjoy the challenge of writing MP code, just as they enjoy the challenge of writing assembler - me too at times ;-) But the pro's and con's of message passing as a programming model, and the alternatives, are not the issues here. While we may have different motivations we do all understand the importance of MP at some level. Certainly it should be usable by "users" in the absence of anything better. In fact I'd contend that my recent calls serve those that do have to use it well - wizard or not. I am not calling for us to take power away from users - on the contrary - I'm calling for greater utility and usability through simplicity. I am calling for us to focus on things we understand well and to avoid invention. I do not feel that such simplicity will fail to serve wizards - your implication is that wizards are only attracted to complexity: this surely is not true. In addition, it is my opinion that there is way too little experience with compiler generated messaging to have such a concept meet the "maturity/experience" criteria upon which we wish to base MPI. No argument here - but then this isn't an objective of MPI. As for overlapped messaging, I dare say that EVERY code here ... I am not saying that non-synchronizing send is a "bad thing" - I am saying there is a simpler way of looking at these things. Define the fundamentals and work from there. We have been arguing about the nuances of buffer behavior without getting a real feel for our base objectives. You can build all you want - if you really want it - with fundamental and well understood components - it makes a great opportunity for third party software vendors like Parasoft and DSL. Buffering characteristics are likely to remain a choice based on hardware design and we should leave such things alone in our interface design. Now I understand that most machine vendors have a variety of approaches - and that is exactly why we should should be careful. But I cannot believe the solution for MPI is the union of those approaches - though it may be some component of the intersection. Isn't our objective to provide a Message Passing Interface that will allow people who want to write portable (or transportable) MP code to do so? And to provide for compiler writers and architects - people like me - who are just looking for a consistent interface on MP machines? Doesn't usability have a right to be at the fore of our concerns? If this is the case then we do need some well conceived abstraction or we must adopt an existing system and refine it. When it comes right down to it, I really don't mind which. I would like though to take a program written in MPI and simply transform it into another MPI program for execution on a machine with different characteristics. What I've seen so far is going to make it very very hard to do that. I'm only likely to want to do literal translation on a machine identical to the original target - it would be nice to permit simple transformations and checking (for deadlock for example) ... wouldn't it? Incidently, in addition to my architects point of view I do still design message passing systems and program them - so I hope I qualify as a user too. ;-) Steven From owner-mpi-pt2pt@CS.UTK.EDU Fri Feb 5 22:04:50 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA17274; Fri, 5 Feb 93 22:04:50 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA13646; Fri, 5 Feb 93 22:04:22 -0500 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Fri, 5 Feb 1993 22:04:21 EST Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from NA-GW.CS.YALE.EDU by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA13638; Fri, 5 Feb 93 22:04:19 -0500 Received: from YOGI.NA.CS.YALE.EDU by CASPER.NA.CS.YALE.EDU via SMTP; Fri, 5 Feb 1993 22:04:17 -0500 Received: by YOGI.NA.CS.YALE.EDU (Sendmail-5.65c/res.client.cf-3.5) id AA05848; Fri, 5 Feb 1993 22:04:16 -0500 Date: Fri, 5 Feb 1993 22:04:16 -0500 From: berryman-harry@CS.YALE.EDU (Harry Berryman) Message-Id: <199302060304.AA05848@YOGI.NA.CS.YALE.EDU> To: mpi-pt2pt@cs.utk.edu Subject: compiler target (was Be brave. Be sure.) Tony says: To follow up on Stephen's comment, I do not believe that compilers would make good use of this standard. Active-message-level programming seems a much better level for standardizing compiler interfaces. There are some compiler folks who might disagree with you. The group at Rice is working on generating message-passing, as well as the group at Syracuse, U. Maryland, and a few other places. Perhaps you could be more specific as to why you think that compilers cannot make good use of this standard? -scott berryman From owner-mpi-pt2pt@CS.UTK.EDU Fri Feb 5 23:21:20 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA17474; Fri, 5 Feb 93 23:21:20 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA15951; Fri, 5 Feb 93 23:20:44 -0500 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Fri, 5 Feb 1993 23:20:43 EST Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from Aurora.CS.MsState.Edu by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA15940; Fri, 5 Feb 93 23:20:42 -0500 Received: by Aurora.CS.MsState.Edu (4.1/6.0s-FWP); id AA01319; Fri, 5 Feb 93 22:20:29 CST Date: Fri, 5 Feb 93 22:20:29 CST From: Tony Skjellum Message-Id: <9302060420.AA01319@Aurora.CS.MsState.Edu> To: mpi-pt2pt@cs.utk.edu, berryman-harry@CS.YALE.EDU Subject: Re: compiler target (was Be brave. Be sure.) Cc: bert@athena.llnl.gov, tsw@mpci.llnl.gov Yes Scott, I can be more specific. I think that MPI[1] is not interesting for compiler writers. Active messages are finer grain message passing, which have lower overhead. There is naturally a larger overhead in a full system call like a send()/recv() pair. If the compiler knows it wants to drop a single real number on a process, it could use a finer grain mechanism. Compilers would be good at cacheing limited open channel resources, such as supported at the low-level by CMMD. Thus, they could take advantage of not having to build up and tear down connections between processes, as is needed for send()/recv(). A good example of this is neighbor communication for boundary sharing in domain decomposition. They could take advantage of special properties of the network(s) of a system, that are lost when a robust send()/recv() mechanism is provided, each of which implies a library [if not system] call. The overhead of a subroutine call send()/recv() implies a minimal granularity, because stack frames must be assembled, etc. The compiler itself could be responsible for the non-deadlock properties of the communication, rather than insisting that the messaging calls provide this capability. Furthermore, compilers could make use of the "force semantics" per Intel, but still provide the user with a guarantee of correctness (ie, use the cheapest calls, but build their own appropriate protocols, but as weak as possible). Mixing communication and computation is another example of where the messaging libraries with send/recv semantics are too heavy duty, compared to active messages. A compiler might want to add two values and store them remotely. It does not need a full stack of messages, with rendezvous to accomplish this. I think the people who are building compilers to generate straight message- passing code should read the work from Berkeley and TMC on active messages (Culler et al). and build compilers that generate code using these excellent ideas. Performance will be superior. Message-passing libraries are for people who write explicit message-passing code without the benefit of compiler support. They need guarantees like non deadlock of messages, they need queues so messages are there when you look for them, and so on. Compilers can manage these things, and avoid the necessary compromises made to present robust message passing to users through send/recv. It is difficult to cope with the many sequential calls that occur in message passing code, without the need to build things like send+recv, in order to reduce overhead. Compilers could optimize away such overheads if they controlled the messaging resource directly (ie, stamp out unneeded overheads). Also, they could inline conversions for heterogeneity, and reduce the number of copies of messages by driving the network as they convert data (or just drive the network as they build parts of a message). Furthermore, they could balance the work done by sender and receiver based on the relative power of the sender and receiver to do conversions. This gets quite complex for hand-written message passing code. Active messaging can also be standardized (eg, create MPI-AM) to provide portable code to machines which are message passing, and which are hierarchically shared memory. So, one could build a model that unifies machines like the Paragon, CM-5, and Cray Tera, for the purpose of compiler code generation, though these are different machines, to be sure. - Tony From owner-mpi-pt2pt@CS.UTK.EDU Mon Feb 8 08:54:05 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA26068; Mon, 8 Feb 93 08:54:05 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA20830; Mon, 8 Feb 93 08:52:54 -0500 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Mon, 8 Feb 1993 08:52:53 EST Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from [129.215.56.21] by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA20822; Mon, 8 Feb 93 08:52:39 -0500 Date: Mon, 8 Feb 93 13:52:17 GMT Message-Id: <16939.9302081352@subnode.epcc.ed.ac.uk> From: L J Clarke Subject: buffered/unbuffered comms (was compiler target (was Be brave. Be sure.)) To: mpi-pt2pt@cs.utk.edu, mpi-comm@cs.utk.edu Reply-To: lyndon@epcc.ed.ac.uk Hi Here comes my 5p worth on the subject of buffered and unbuffered comms. For completeness: by unbuffered I mean that the sender blocks until the message has been (or certainly will be) copied into the space of the receiver, a la occam for example; by buffered comms I mean that the message is copied away into a system buffer somewhere and the sender returns, a la PVM for example. It seems to me that there are a few different issues in this subject which discussions may have touched on. a) Ease of programming. I don't think anyone can really disagree that many programmers will report that they find buffered comms easier to use. These users have been fortunate enough not to have come up against the boundedness of buffering provided with the system. With high probability they are programming (or adapting) applications by inserting message passing calls directly into the application source. b) Portability of programs using MPI between different machines. [Here I give away my bias.] It is very difficult to make statements about portability (and reliability/correctness) of programs which use buffered messages and rely on system storage capacity and structure thereof. This becomes particularly difficult when the application makes use of substantive libraries, which themselves are written using buffered comms and place requirements on system buffering. It can just become to difficult to work out how much buffering you need. We played with a system which allowed the programmer to configure the system buffering, and this feature was only used (properly) by high priests. In my work I will only be able to use MPI if I can write substantive libraries in the confidence that they will not be subject to failure due to running out of such buffer space. Therefore we use only unbuffered message passing with a mixture of blocking/nonblocking (like Intel NX isend/irecv/msgwait) calls. Therefore my bias :-) c) Portability of existing applications (using existing message passing interfaces) to MPI. This is a different subject to (c). Following the discussion it seems that an argument in favour of adopting buffered comms is the number of existing programs (eg, lotsa programs written using Express) which use unbuffered comms. It has been my experience that migration of applications between different, broadly similar, message passing interfaces is not so difficult, except for this point. In a nutshell, the surgery you have to perform to move programs/libraries between an interface that forces buffered comms and one which forces unbuffered comms and blocking/nonblocking (isend/irecv/msgwait) is grevious and error prone. Given the above, I come to the conclusions: i) MPI must contain unbuffered communications with blocking/nonblocking (irecv/isend/msgwait kinda thing) calls, for reliability and portability. ii) If a goal of MPI is that existing applications using message passing interfaces (eg Express, PVM) should easily port to MPI, then MPI must also contain buffered comms. This seems to be a matter for the full committee, hence I have crossposted. Incidentally, to pitch in 2ps on some other lines of discussion: * I can see no particular benefit in allowing a buffered snd/rcv match an unbuffered rcv/snd. I can see considerable inconvenience in forbidding a blocking send/recv to match with a nonblocking recv/send. * Yes, we do try to overlap communications with calculation. (Sometimes it doesnt buy you, sometimes it does. Sometimes we have specifically needed to avoid such overlap in order to maximise performance, but that was a weird one :-) Just as important, we frequently "overlap" communication with communication (ie, use nonblocking calls) to avoid deadlock. * Please, oh please, let MPI decide that communications should be adressed using a rank relative to a group/context (0 ... GroupSize - 1) or (1 ... GroupSize). We do this all the time, and its very very convenient. In fact, when we can't do this we end up having to create arrays of the task identifiers - we end up doing it ourselves anyway. Best Wishes Lyndon /--------------------------------------------------------\ e||) | Lyndon J Clarke Edinburgh Parallel Computing Centre | e||) c||c | Tel: 031 650 5021 Email: lyndon@epcc.edinburgh.ac.uk | c||c \--------------------------------------------------------/ From owner-mpi-pt2pt@CS.UTK.EDU Mon Feb 8 09:25:16 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA27616; Mon, 8 Feb 93 09:25:16 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA22596; Mon, 8 Feb 93 09:24:49 -0500 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Mon, 8 Feb 1993 09:24:46 EST Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from watson.ibm.com by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA22575; Mon, 8 Feb 93 09:24:40 -0500 Message-Id: <9302081424.AA22575@CS.UTK.EDU> Received: from YKTVMV by watson.ibm.com (IBM VM SMTP V2R2) with BSMTP id 9817; Mon, 08 Feb 93 09:24:38 EST Date: Mon, 8 Feb 93 09:24:37 EST From: "Marc Snir" To: MPI-PT2PT@CS.UTK.EDU Received: from YKTVMH by watson.vnet.ibm.com with "VAGENT.V1.0" id 7422; Mon, 8 Feb 1993 09:19:14 EST Received: from snir.watson.ibm.com by yktvmh.watson.ibm.com (IBM VM SMTP V2R2) with TCP; Mon, 08 Feb 93 09:19:11 EST Received: by snir.watson.ibm.com (AIX 3.2/UCB 5.64/930113) id AA24152; Mon, 8 Feb 1993 09:19:15 -0500 Date: Mon, 8 Feb 1993 09:19:15 -0500 From: snir@snir.watson.ibm.com (Marc Snir) Message-Id: <9302081419.AA24152@snir.watson.ibm.com> To: snir@yktvmv.vnet.ibm.com \documentstyle[12pt]{article} \newcommand{\discuss}[1]{ \ \\ \ \\ {\small {\bf Discussion:} #1} \\ \ \\ } \newcommand{\missing}[1]{ \ \\ \ \\ {\small {\bf Missing:} #1} \\ \ \\ } \begin{document} \title{ Point to Point Communication } \author{Marc Snir} \maketitle \section{Point to Point Communication} \subsection{Introduction} This section is a draft of the current proposal for point-to-point communication. We have not discussed in our last meeting the choice of a syntax for the various communication operations. In particular, there have been no discussion on the choice of function names. There has been no decision whether its preferable to use distinct function names or use additional parameters to distinguish between different variants of the same operation. There has been no discussion whether parameters should be kept separated or bundled in one ``capsule'' that is passed as one parameter. There has been no discussion whether we prefer, in Fortran, functions or subroutines. Thus, the choice of a particular syntax in this draft is merely for the sake of exposition, and is subject to change. I have tried to indicate, wherever appropriate, gaps and unresolved issues, using small type. The current draft borrows heavily on the proposal of Gropp and Lusk for a Multi-Level Message Passing Interface and on my previous outline. \subsection{Messages} A message consists of an {\em envelope} and {\em data}. \subsubsection{Data} \paragraph*{Untyped Data} The simplest (and most used) message data is untyped. Such data consists of a sequence of bytes. Untyped messages can be used to transfer arbitrary typed values between processors with identical data representations; thus, untyped messages are sufficient in a homogeneous environment, where no type conversion is required. \discuss{ Is it legitimate to require that message length be a multiple of 4 or 8? -- or should we require that any number of bytes be OK, with communication perhaps faster for multiples of 4 or 8?} \paragraph*{Typed data} \missing{Need to add here typed messages.} \subsubsection{Envelope} \label{subsec:envelope} The following information is associated with each message: \begin{description} \item[source] The id of the sending task \item[dest] The id of the receiving task \item[tag] User defined \item[context] \end{description} All these fields are of type integer. The range of valid values for the {\bf sender} and {\bf receiver} fields is {\tt 0 ... number\_of\_tasks-1}, where {\tt number\_of\_tasks} is the number of tasks currently participating in the computation (i.e., both fields must include a valid task number). The ranges of valid values for {\tt tag} and {\tt context} are implementation dependent, and can be found by calling a suitable query function, as described in Section~\ref{sec:inquiry}. \discuss{ Users would prefer to be able to use any integer as {\tt tag} or {\tt context}. Vendors may prefer small ranges (e.g. 8 or 16 bits) so as to use fast tag and/or context matching hardware. A defined arithmetic type could be used in C for {\tt tag} and {\tt context}. } The {\tt tag} field can be arbitrarily set by the application, and can be used to distinguish different messages. The {\tt context} field is used to distinguish messages produced by different applications or different libraries. Consider the case where an application calls in parallel on all executing nodes a library function. One needs to make sure that a message generated by the library code will not be consumed by the application and vice-versa. To do so, one can \begin{enumerate} \item Synchronize all nodes before and after the call, or \item Follow a programming convention that prevents such confusion (e.g., make sure that the library code generate messages with tags that are distinct from the tags used by the application, and accepts only messages with correct tags). \end{enumerate} The first solution may lead to less efficient code (it prevents a ``loosely synchronous'' library call). The second solution prevents modular code development. The use of a {\tt context} field, together with the context setting mechanisms described in Section~\ref{sec:context}, solve this problem. Finally, we would like to note that the actual mechanism used to associate data with envelope is implementation dependent; some of the information (e.g., {\bf sender} or {\bf receiver}) may be implicit in some context, and may not be explicitly carried by a message. \subsection{Data Buffers} \label{subsec:buffers} The basic point to point communication operations are {\bf send} and {\bf receive}. A {\bf send} operation creates a message; the message data is assembled from the {\bf send buffer}. A {\bf receive} operation consumes a message; the message data is moved into the {\bf receive buffer}. The specification of send or receive buffers uses the same syntax. \paragraph*{Untyped Messages} There are three kinds of buffers for untyped messages. \begin{description} \item[contiguous buffer] A sequence of contiguous bytes in memory, specified by \begin{description} \item[buf] Initial address \item[len] Number of bytes \end{description} \item[constant stride buffer] A sequence of blocks equally spaced and equally sized blocks, specified by \begin{description} \item[buf] Initial address \item[numblk] Number of blocks \item[lenblk] Number of bytes in each block \item[stride] Number of bytes between the start of each block \end{description} Note that a constant stride buffer becomes a contiguous buffer when {\tt stride = lenblk}. \item[general scatter gather] A sequence of constant stride buffers, specified by a {\bf data\_vector} of the form {\tt (number\_of\_items, buf1, numblk1, lenblk1, stride1, ... bufn, numblkn, lenblkn, striden)}. \end{description} \discuss{ Since Fortran 77 does not have a pointer type, an MPI library that is strictly Fortran 77 compliant would need a different send routine for each possible type of the variable {\tt buf}. This is not very practical. I suggest to ignore the issue and rely on the fact that Fortran 77 compilers and linkers are very unlikely to do type checking across separately compiled modules and are passing routine parameters by reference. The C binding can use pointers, and the Fortran 90 binding can use overloaded subroutines (for basic types). Data vectors are more of a problem, since we now need pointers as components of the data vector. The Posix Fortran 77 Language Interface IEEE standard does not provide a general enough mechanism for that purpose. We might restrict the use of data vectors to C. (Of course, sophisticated programmers would be able to call C routines from Fortran, and nonstandard solutions might also be provided.) Or, we might require that all buffer addresses are integer displacements with respect to the first address, in which case all entries are integer. Is it acceptable to have a (small) fixed upper bound on the length of data vectors? } \paragraph*{Typed Buffers} \missing{Data buffers for typed messages.} \subsection{Receive Criteria} The selection of a message by a receive operation is done according to the value of the message envelope. The receive operation specifies an {\bf envelope pattern}; a message can be received by that receive operation only if its envelope matches that pattern. A pattern specifies values for the {\tt source}, {\tt tag} and {\tt context} fields of the message envelope. In addition, the value for the {\tt dest} field is set, implicitly, to be equal to the receiving process id. The receiver may specify a {\bf DONTCARE} value for {\tt source}, or {\tt tag}, indicating that any source and/or tag are acceptable. It cannot specify a DONTCARE value for {\tt context} or {\tt dest}. Thus, a message can be received by a receive operation only if it is addressed to the receiving task, has a matching context, has matching source unless source=DONTCARE in the pattern, and has a matching tag unless tag=DONTCARE in the pattern. The length of the received message must be less or equal the length of the receive buffer. I.e., all incoming data must fit, without truncation, into the receive buffer. It is erroneous to receive a message which length exceed the receive buffer, and the outcome of program where this occurs is undetermined. \discuss{ The DONTCARE value can be a fixed value (e.g. -1), and/or a named constant. It might be useful to have the option to require exact size match of incoming message and receive buffer. Some would also like to see truncation supported. } \subsection{Communication Handles} One can consider (most) communication operations to consist of the following suboperations: \begin{description} \item[INIT(operation, params, handle)] Process provides all relevant parameters for its participation in the communication operation (type of operation, data buffer, tag, participants, etc.). A handle is created that identifies the operation. \item[START(handle)] The communication operation is started \item[COMPLETE(handle)] The communication operation is completed. \item[FREE(handle)] The handle, and associated resources are freed. \end{description} Correct invocation of these suboperations is a sequence of the form \[ \bf INIT \ (START \ COMPLETE )^* \ FREE .\] I.e., a handle needs be created before communication occurs; it can be reused only after the previous use has completed; and it needs to be freed eventually (of course, one can assume that all handles are freed at program termination, by default). A user may directly invokes these suboperations. This would allow to amortize the overhead of setting up a communication over many successive uses of the same handle, and allows to overlap communication and computation. Simpler communication operations combine several of these suboperations into one operation, thus simplifying the use of communication primitives. Thus, one only needs to specify precisely the semantics of these suboperations in order to specify the semantics of MPI point to point communication operations. We say that a communication operation (send or receive) is {\bf posted} once a {\bf start} suboperation was invoked; the operation is {\bf completed} once the {\bf complete} suboperation completes. A send and a receive operation {\bf match} if the receive pattern specified by the receive matches the message envelope created by the send. \subsubsection{Handle Creation} A handle for a send operation is created by a call to {\bf MPI\_INIT\_SEND}, followed by a set of calls that set the various parameters associated with the handle. The handle cannot be used until all parameters have been set. A call to {\bf MPI\_INIT\_RECV} is similarly used for creating a handle for a receive operation. Handle creation is a local operation that need not involve communication with a remote process. {\bf handle = MPI\_INIT\_SEND (dest, tag, context)} \\ Create a send handle. All three parameters must be specified. {\bf handle = MPI\_INIT\_RECV (source, tag, context)} \\ Create a send handle. DONTCARE values may be used for the first two parameters. See Section~\ref{subsec:envelope} for a discussion of source, tag and context. \discuss{ An alternative design, proposed by Gropp and Lusk, is to have each newly created handle associated with default parameters that can be later modified. In their proposal, too, all modifications must occur before the first use.} \subsubsection{Buffer specification} The {\bf MPI\_BUFFER} function associates a buffer with a handle. {\bf MPI\_BUFFER(handle, buftype, params)}\\ \begin{tabbing} {\bf buftype} \ \ \ \ \ \ \ \ \ \ \ \= {\bf params} \\ CONTIGUOUS \> buf, len \\ STRIDED \> buf, numblk, lenblk, stride \\ GENERAL \> data\_vector \end{tabbing} See Section~\ref{subsec:buffers} for a description of these buffer types and parameters. The {\tt buftype} parameter is an integer, with named constants provided for the three values. The {\tt params} parameter list consists of an address (pointer) for {\tt buf}, and a one-dimensional array of integers to specify the remaining values. See Section~\ref{subsec:buffers} for a discussion of data vectors. \missing{typed buffers} \discuss{ Some may want specific values for {\tt buftype}, e.g. CONTIGUOUS =0, STRIDED=1, GENERAL=2. } \subsubsection{Start Mode} The {\bf MPI\_START\_MODE} function selects a mode for starting a communication operation. {\bf MPI\_START\_MODE(mode)} The mode parameter is an integer, with named constants provided for its two values. There are two modes: \begin{description} \item[REGULAR] The operation may start whether or not a matching communication operation has been posted. \item[READY] The operation my start only if a matching communication operation has been posted. \end{description} Thus, a {\bf ready send} can start only if a matching receive is already posted; otherwise the operation is erroneous and its outcome is undefined. A {\bf ready receive} can start only if a matching send is already posted; otherwise the operation is erroneous and its outcome is undefined. \discuss{ May want to fix the values, e.g., REGULAR=0 and READY=1. We discussed only ready send, not ready receive. Ready send is useful when message passing is implemented in a ``push'' style, where the sender moves the message to the receiver (common case). Ready receive would be useful when message passing is implemented in a ``pull'' style, where the receiver moves the data from the sender. } \subsubsection{Completion Mode} \label{subsec:completion} The {\bf MPI\_COMPLETE\_MODE} function selects a mode for completing a communication operation. {\bf MPI\_COMPLETE\_MODE(mode)} The mode parameter is an integer, with named constants provided for its two values. There are two modes: \begin{description} \item[REGULAR] The operation completes locally, irrespective of the completion of the matching operation. \item[SYNCHRONOUS] The operation completes only if the matching operation completes too. \end{description} A regular send can complete as soon as the message has been composed: data has been copied out of the send buffer and the values used for the envelope have been stored. The sender may now update the send buffer. A regular send can complete before the matching receive has completed. A regular receive can complete as soon as the message has been consumed: data is available in the receive buffer and all values returned by the send complete operation are available. The receiver may now access the receive buffer. A regular receive may complete before the matching send has completed (It may seem strange, at first blush, that a receive may complete before the matching send completes. However, one should keep in mind that an operation completes only when the {\tt complete} suboperation terminates. Thus, even though data has been copied out and the message was sent, a send has not not yet completed if the {\tt send complete} suboperation has not yet been invoked. Informally, the send (receive) completes only when the sender (receiver) knows the operation has completed.) A synchronous communication operation completes only if the matching operation completes too. Thus, a synchronous send can complete as soon as the all operations preceding the completion of the matching receive have terminated. A synchronous receive may terminate as soon as the message has been consumed and all operations preceding the completion of the matching send have terminated. Informally, the completion of the matching send and receive operations are synchronized. Matching communication operations must have the same completion mode. I.e., either both send and receive are regular or both are synchronous. A program where a regular receive matches a synchronous send (or vice versa) is erroneous and its outcome is undefined. \discuss{ May want to fix the values, e.g., REGULAR=0 and SYNCHRONOUS=1. } \subsubsection{Communication Start} {\bf MPI\_START(handle)} \\ The {\tt MPI\_START} function starts the execution of a communication operation (send or receive). A sender should not update the send buffer after a send operation has started and until it is completed. A receiver should not access the receive buffer after a receive operation was started and until it is completed. A program that does not satisfy this condition is erroneous and its outcome is undetermined. \discuss{ Some might prefer two distinct functions: {\bf MPI\_SEND\_START} and {\bf MPI\_RECV\_START}. } \subsubsection{Communication Completion} \label{subsec:complete_ops} {\bf MPI\_COMPLETE\_SEND ( handle)} \\ A call to {\bf MPI\_COMPLETE\_SEND} returns when the send operation identified by {\tt handle} is complete (see Section~\ref{subsec:completion} for a description of completion criteria). {\bf MPI\_COMPLETE\_RECV ( handle, len, source, tag)} \\ A call to {\bf MPI\_COMPLETE\_RECV} returns when the receive operation identified by {\tt handle} is complete (see Section~\ref{subsec:completion} for a description of completion criteria). The call returns the number of bytes received, and the source and tag of the message received. {\bf MPI\_CHECK\_SEND ( handle, flag)} \\ A call to {\bf MPI\_CHECK\_SEND} returns {\tt flag=true} if the send operation identified by {\tt handle} is complete (see Section~\ref{subsec:completion} for a description of completion criteria). It returns {\tt flag=false}, otherwise. A successful return of {\bf MPI\_CHECK\_SEND} has the same semantics as a return of {\bf MPI\_COMPLETE\_SEND}. {\bf MPI\_CHECK\_RECV ( handle, flag, len, source, tag)} \\ A call to {\bf MPI\_CHECK\_RECV} returns {\tt flag=true} if the receive operation identified by {\tt handle} is complete (see Section~\ref{subsec:completion} for a description of completion criteria). In such case the call returns the number of bytes received, and the source and tag of the message received. The call returns {\tt flag=false}, otherwise. In such case, the return values of {\tt len}, {\tt source} and {\tt tag} are undefined. Implementation notes: A call to {\tt MPI\_COMPLETE} blocks only the executing thread. If the executing process is multithreaded, then other threads within the process can be scheduled for execution. A call to {\tt MPI\_COMPLETE\_SEND(handle)} has the same semantics as a busy loop of the form \begin{verbatim} repeat MPI_CHECK_SEND(handle, flag) until(flag=true) \end{verbatim} and similarly for receives. The use of a blocking receive operation ({\tt MPI\_COMPLETE}) allows the operating system to deschedule the blocked thread and schedule another thread for execution, if such is available. The use of a nonblocking receive operation ({\tt MPI\_CHECK}) allows the user to schedule alternative activities within a single thread of execution. The intended implementation of {\tt MPI\_CHECK} is for that operation to return as soon as possible. Note that it is correct, but inefficient, to implement {\tt MPI\_CHECK} via a call to {\tt MPI\_COMPLETE}, in which case, {\tt MPI\_CHECK} always returns {\tt true}. \discuss{ One might use {\tt len=-1} to indicate unsuccessful return of {\tt MPI\_CHECK\_RECV} } \subsection{Blocking Communication} Blocking send and receive operations combine all four suboperations into one call. The operation returns only when the communication completes and no communication handle persists after the call completed. We use the following naming convention for such operations: \[ \left[ \begin{array}{c} - \\ \bf r \end{array} \right] \left[ \begin{array}{c} - \\ \bf s \end{array} \right] \left[ \begin{array}{c} \bf send \\ \bf recv \end{array} \right] \left[ \begin{array}{c} - \\ \bf s \\ \bf g \end{array} \right] \] The first letter (void or {\bf r}) indicates the start mode (regular or ready). The second letter (void or {\bf s}) indicates the completion mode (regular or synchronous). The last letter (void, {\bf s} or {\bf g}) indicates the buffer type (contiguous, strided or general). The corresponding 12 send and 12 receive operations are listed below. {\bf MPI\_SEND(buf, len, dest, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_SEND(dest, tag, context) MPI_BUFFER(handle, CONTIGUOUS, (len)) MPI_START_MODE(handle, REGULAR) MPI_COMPLETE_MODE(handle, REGULAR) MPI_START(handle) MPI_COMPLETE_SEND(handle) MPI_FREE(handle) \end{verbatim} {\bf MPI\_SENDS(buf, numblk, lenblk, stride, dest, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_SEND(dest, tag, context) MPI_BUFFER(handle, STRIDED, buf, (numblk, lenblk, stride)) MPI_START_MODE(handle, REGULAR) MPI_COMPLETE_MODE(handle, REGULAR) MPI_START(handle) MPI_COMPLETE_SEND(handle) MPI_FREE(handle) \end{verbatim} {\bf MPI\_SENDG(vector, dest, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_SEND(dest, tag, context) MPI_BUFFER(handle, GENERAL, (vector)) MPI_START_MODE(handle, REGULAR) MPI_COMPLETE_MODE(handle, REGULAR) MPI_START(handle) MPI_COMPLETE_SEND(handle) MPI_FREE(handle) \end{verbatim} {\bf MPI\_SSEND(buf, len, dest, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_SEND(dest, tag, context) MPI_BUFFER(handle, CONTIGUOUS, (len)) MPI_START_MODE(handle, REGULAR) MPI_COMPLETE_MODE(handle, SYNCHRONOUS) MPI_START(handle) MPI_COMPLETE_SEND(handle) MPI_FREE(handle) \end{verbatim} {\bf MPI\_SSENDS(buf, numblk, lenblk, stride, dest, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_SEND(dest, tag, context) MPI_BUFFER(handle, STRIDED, buf, (numblk, lenblk, stride)) MPI_START_MODE(handle, REGULAR) MPI_COMPLETE_MODE(handle, SYNCHRONOUS) MPI_START(handle) MPI_COMPLETE_SEND(handle) MPI_FREE(handle) \end{verbatim} {\bf MPI\_SSENDG(vector, dest, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_SEND(dest, tag, context) MPI_BUFFER(handle, GENERAL, (vector)) MPI_START_MODE(handle, REGULAR) MPI_COMPLETE_MODE(handle, SYNCHRONOUS) MPI_START(handle) MPI_COMPLETE_SEND(handle) MPI_FREE(handle) \end{verbatim} {\bf MPI\_RSEND(buf, len, dest, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_SEND(dest, tag, context) MPI_BUFFER(handle, CONTIGUOUS, (len)) MPI_START_MODE(handle, READY) MPI_COMPLETE_MODE(handle, REGULAR) MPI_START(handle) MPI_COMPLETE_SEND(handle) MPI_FREE(handle) \end{verbatim} {\bf MPI\_RSENDS(buf, numblk, lenblk, stride, dest, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_SEND(dest, tag, context) MPI_BUFFER(handle, STRIDED, buf, (numblk, lenblk, stride)) MPI_START_MODE(handle, READY) MPI_COMPLETE_MODE(handle, REGULAR) MPI_START(handle) MPI_COMPLETE_SEND(handle) MPI_FREE(handle) \end{verbatim} {\bf MPI\_RSENDG(vector, dest, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_SEND(dest, tag, context) MPI_BUFFER(handle, GENERAL, (vector)) MPI_START_MODE(handle, READY) MPI_COMPLETE_MODE(handle, REGULAR) MPI_START(handle) MPI_COMPLETE_SEND(handle) MPI_FREE(handle) \end{verbatim} {\bf MPI\_RSSEND(buf, len, dest, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_SEND(dest, tag, context) MPI_BUFFER(handle, CONTIGUOUS, (len)) MPI_START_MODE(handle, READY) MPI_COMPLETE_MODE(handle, SYNCHRONOUS) MPI_START(handle) MPI_COMPLETE_SEND(handle) MPI_FREE(handle) \end{verbatim} {\bf MPI\_RSSENDS(buf, numblk, lenblk, stride, dest, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_SEND(dest, tag, context) MPI_BUFFER(handle, STRIDED, buf, (numblk, lenblk, stride)) MPI_START_MODE(handle, READY) MPI_COMPLETE_MODE(handle, SYNCHRONOUS) MPI_START(handle) MPI_COMPLETE_SEND(handle) MPI_FREE(handle) \end{verbatim} {\bf MPI\_RSSENDG(vector, dest, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_SEND(dest, tag, context) MPI_BUFFER(handle, GENERAL, (vector)) MPI_START_MODE(handle, READY) MPI_COMPLETE_MODE(handle, SYNCHRONOUS) MPI_START(handle) MPI_COMPLETE_SEND(handle) MPI_FREE(handle) \end{verbatim} {\bf MPI\_RECV(buf, len, source, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_RECV(source, tag, context) MPI_BUFFER(handle, CONTIGUOUS, (len)) MPI_START_MODE(handle, REGULAR) MPI_COMPLETE_MODE(handle, REGULAR) MPI_START(handle) MPI_COMPLETE_SEND(handle,len,source,tag) MPI_FREE(handle) \end{verbatim} {\bf MPI\_RECVS(buf, numblk, lenblk, stride, len, source, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_RECV(source, tag, context) MPI_BUFFER(handle, STRIDED, buf, (numblk, lenblk, stride)) MPI_START_MODE(handle, REGULAR) MPI_COMPLETE_MODE(handle, REGULAR) MPI_START(handle) MPI_COMPLETE_RECV(handle, len, source, tag) MPI_FREE(handle) \end{verbatim} {\bf MPI\_RECVG(vector, len, source, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_RECV(source, tag, context) MPI_BUFFER(handle, GENERAL, (vector)) MPI_START_MODE(handle, REGULAR) MPI_COMPLETE_MODE(handle, REGULAR) MPI_START(handle) MPI_COMPLETE_RECV(handle, len, source, tag) MPI_FREE(handle) \end{verbatim} {\bf MPI\_SRECV(buf, len, source, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_RECV(source, tag, context) MPI_BUFFER(handle, CONTIGUOUS, (len)) MPI_START_MODE(handle, REGULAR) MPI_COMPLETE_MODE(handle, SYNCHRONEOUS) MPI_START(handle) MPI_COMPLETE_RECV(handle, len, source, tag) MPI_FREE(handle) \end{verbatim} {\bf MPI\_SRECVS(buf, numblk, lenblk, stride, len, source, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_RECV(source, tag, context) MPI_BUFFER(handle, STRIDED, buf, (numblk, lenblk, stride)) MPI_START_MODE(handle, REGULAR) MPI_COMPLETE_MODE(handle, SYNCHRONOUS) MPI_START(handle) MPI_COMPLETE_RECV(handle, len, source, tag) MPI_FREE(handle) \end{verbatim} {\bf MPI\_SRECVG(vector, len, source, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_RECV(source, tag, context) MPI_BUFFER(handle, GENERAL, (vector)) MPI_START_MODE(handle, REGULAR) MPI_COMPLETE_MODE(handle, SYNCHRONEOUS) MPI_START(handle) MPI_COMPLETE_RECV(handle, len, source, tag) MPI_FREE(handle) \end{verbatim} {\bf MPI\_RRECV(buf, len, source, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_RECV(source, tag, context) MPI_BUFFER(handle, CONTIGUOUS, (len)) MPI_START_MODE(handle, READY) MPI_COMPLETE_MODE(handle, REGULAR) MPI_START(handle) MPI_COMPLETE_RECV(handle, len, source, tag) MPI_FREE(handle) \end{verbatim} {\bf MPI\_RRECVS(buf, numblk, lenblk, stride, len, source, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_RECV(source, tag, context) MPI_BUFFER(handle, STRIDED, buf, (numblk, lenblk, stride)) MPI_START_MODE(handle, READY) MPI_COMPLETE_MODE(handle, REGULAR) MPI_START(handle) MPI_COMPLETE_RECV(handle) MPI_FREE(handle) \end{verbatim} {\bf MPI\_RRECVG(vector, len, source, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_RECV(source, tag, context) MPI_BUFFER(handle, GENERAL, (vector)) MPI_START_MODE(handle, READY) MPI_COMPLETE_MODE(handle, REGULAR) MPI_START(handle) MPI_COMPLETE_RECV(handle, len, soure, tag) MPI_FREE(handle) \end{verbatim} {\bf MPI\_RSRECV(buf, len, source, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_RECV(source, tag, context) MPI_BUFFER(handle, CONTIGUOUS, (len)) MPI_START_MODE(handle, READY) MPI_COMPLETE_MODE(handle, SYNCHRONOUS) MPI_START(handle) MPI_COMPLETE_RECV(handle, len, source, tag) MPI_FREE(handle) \end{verbatim} {\bf MPI\_RSRECVS(buf, numblk, lenblk, stride, len, source, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_RECV(source, tag, context) MPI_BUFFER(handle, STRIDED, buf, (numblk, lenblk, stride)) MPI_START_MODE(handle, READY) MPI_COMPLETE_MODE(handle, SYNCHRONEOUS) MPI_START(handle) MPI_COMPLETE_RECV(handle, len, source, tag) MPI_FREE(handle) \end{verbatim} {\bf MPI\_RSRECVG(vector, len, source, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_RECV(source, tag, context) MPI_BUFFER(handle, GENERAL, (vector)) MPI_START_MODE(handle, READY) MPI_COMPLETE_MODE(handle, SYNCHRONOUS) MPI_START(handle) MPI_COMPLETE_RECV(handle, len, source, tag) MPI_FREE(handle) \end{verbatim} Note that all receive calls return the actual length, tag and source of the received message. This, the {\tt source} and {\tt tag} parameters are INOUT. The {\tt len} parameter is INOUT for contiguous receive buffers, OUT for strided and general receive buffers. {\bf Implementation note:} While these 24 functions can be implemented via calls to functions that implement suboperations, as described in this subsection, an efficient implementation may optimize away these multiple calls, provided it does not change the behavior of correct programs. \discuss{ Not all 24 functions are equally important (ready-synchronous is somewhat esoteric). We can decide to organize the functins so that the most important are in a ``core subset''. But (i) users will use whatever you provide, and will find uses for what you believed to be useless; (ii) there is a virtue in being systematic -- no need to remember what works and what does not; (iii) what is not important can be implemented using multiple calls to the basic functions. The need for an additional {\tt len} parameter for strided and general receives is somewhat annoying. Suggestions? } \subsection{Nonblocking Communication} Nonblocking send and receive operations combine the first two suboperations ({\tt INIT} and {\tt START}) into one call, and the last two suboperations ({\tt COMPLETE} or {\tt CHECK} and {\tt FREE}) into another suboperation. Thus a nonblocking send or receive consists of two suboperations: One that starts the communication, and one that completes the operation and free any associated resources. No communication handle persists after these two suboperations are executed. \subsubsection{Initiation} We use the following naming convention for the first suboperation of a nonblocking communication: \[ \left[ \begin{array}{c} - \\ \bf r \end{array} \right] \left[ \begin{array}{c} - \\ \bf s \end{array} \right] \left[ \begin{array}{c} \bf isend \\ \bf irecv \end{array} \right] \left[ \begin{array}{c} - \\ \bf s \\ \bf g \end{array} \right] \] The first letter (void or {\bf r}) indicates the start mode (regular or ready). The second letter (void or {\bf s}) indicates the completion mode (regular or synchronous). The last letter (void, {\bf s} or {\bf g}) indicates the buffer type (contiguous, strided or general). The corresponding 12 send and 12 receive operations are listed below. The letter {\bf i} in {\bf isend} and {\bf irecv} indicates that the operation is ``immediate'' and that the call may return before the operation is completed. We thus have 12 immediate send operations and 12 immediate receive operations. {\bf MPI\_ISEND(buf, len, dest, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_ISEND(dest, tag, context) MPI_BUFFER(handle, CONTIGUOUS, (len)) MPI_START_MODE(handle, REGULAR) MPI_COMPLETE_MODE(handle, REGULAR) MPI_START(handle) \end{verbatim} {\bf MPI\_ISENDS(buf, numblk, lenblk, stride, dest, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_SEND(dest, tag, context) MPI_BUFFER(handle, STRIDED, buf, (numblk, lenblk, stride)) MPI_START_MODE(handle, REGULAR) MPI_COMPLETE_MODE(handle, REGULAR) MPI_START(handle) \end{verbatim} {\bf MPI\_ISENDG(vector, dest, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_SEND(dest, tag, context) MPI_BUFFER(handle, GENERAL, (vector)) MPI_START_MODE(handle, REGULAR) MPI_COMPLETE_MODE(handle, REGULAR) MPI_START(handle) \end{verbatim} {\bf MPI\_SISEND(buf, len, dest, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_SEND(dest, tag, context) MPI_BUFFER(handle, CONTIGUOUS, (len)) MPI_START_MODE(handle, REGULAR) MPI_COMPLETE_MODE(handle, SYNCHRONOUS) MPI_START(handle) \end{verbatim} {\bf MPI\_SISENDS(buf, numblk, lenblk, stride, dest, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_SEND(dest, tag, context) MPI_BUFFER(handle, STRIDED, buf, (numblk, lenblk, stride)) MPI_START_MODE(handle, REGULAR) MPI_COMPLETE_MODE(handle, SYNCHRONOUS) MPI_START(handle) \end{verbatim} {\bf MPI\_SISENDG(vector, dest, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_SEND(dest, tag, context) MPI_BUFFER(handle, GENERAL, (vector)) MPI_START_MODE(handle, REGULAR) MPI_COMPLETE_MODE(handle, SYNCHRONEOUS) MPI_START(handle) \end{verbatim} {\bf MPI\_RISEND(buf, len, dest, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_SEND(dest, tag, context) MPI_BUFFER(handle, CONTIGUOUS, (len)) MPI_START_MODE(handle, READY) MPI_COMPLETE_MODE(handle, REGULAR) MPI_START(handle) \end{verbatim} {\bf MPI\_RISENDS(buf, numblk, lenblk, stride, dest, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_SEND(dest, tag, context) MPI_BUFFER(handle, STRIDED, buf, (numblk, lenblk, stride)) MPI_START_MODE(handle, READY) MPI_COMPLETE_MODE(handle, REGULAR) MPI_START(handle) \end{verbatim} {\bf MPI\_RISENDG(vector, dest, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_SEND(dest, tag, context) MPI_BUFFER(handle, GENERAL, (vector)) MPI_START_MODE(handle, READY) MPI_COMPLETE_MODE(handle, REGULAR) MPI_START(handle) \end{verbatim} {\bf MPI\_RSISEND(buf, len, dest, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_SEND(dest, tag, context) MPI_BUFFER(handle, CONTIGUOUS, (len)) MPI_START_MODE(handle, READY) MPI_COMPLETE_MODE(handle, SYNCHRONEOUS) MPI_START(handle) \end{verbatim} {\bf MPI\_RSISENDS(buf, numblk, lenblk, stride, dest, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_SEND(dest, tag, context) MPI_BUFFER(handle, STRIDED, buf, (numblk, lenblk, stride)) MPI_START_MODE(handle, READY) MPI_COMPLETE_MODE(handle, SYNCHRONOUS) MPI_START(handle) \end{verbatim} {\bf MPI\_RSISENDG(vector, dest, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_SEND(dest, tag, context) MPI_BUFFER(handle, GENERAL, (vector)) MPI_START_MODE(handle, READY) MPI_COMPLETE_MODE(handle, SYNCHRONOUS) MPI_START(handle) \end{verbatim} {\bf MPI\_IRECV(buf, len, source, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_RECV(source, tag, context) MPI_BUFFER(handle, CONTIGUOUS, (len)) MPI_START_MODE(handle, REGULAR) MPI_COMPLETE_MODE(handle, REGULAR) MPI_START(handle) \end{verbatim} {\bf MPI\_IRECVS(buf, numblk, lenblk, stride, len, source, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_RECV(source, tag, context) MPI_BUFFER(handle, STRIDED, buf, (numblk, lenblk, stride)) MPI_START_MODE(handle, REGULAR) MPI_COMPLETE_MODE(handle, REGULAR) MPI_START(handle) \end{verbatim} {\bf MPI\_IRECVG(vector, len, source, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_RECV(source, tag, context) MPI_BUFFER(handle, GENERAL, (vector)) MPI_START_MODE(handle, REGULAR) MPI_COMPLETE_MODE(handle, REGULAR) MPI_START(handle) \end{verbatim} {\bf MPI\_SIRECV(buf, len, source, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_RECV(source, tag, context) MPI_BUFFER(handle, CONTIGUOUS, (len)) MPI_START_MODE(handle, REGULAR) MPI_COMPLETE_MODE(handle, SYNCHRONOUS) MPI_START(handle) \end{verbatim} {\bf MPI\_SIRECVS(buf, numblk, lenblk, stride, len, source, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_RECV(source, tag, context) MPI_BUFFER(handle, STRIDED, buf, (numblk, lenblk, stride)) MPI_START_MODE(handle, REGULAR) MPI_COMPLETE_MODE(handle, SYNCHRONOUS) MPI_START(handle) \end{verbatim} {\bf MPI\_SIRECVG(vector, len, source, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_RECV(source, tag, context) MPI_BUFFER(handle, GENERAL, (vector)) MPI_START_MODE(handle, REGULAR) MPI_COMPLETE_MODE(handle, SYNCHRONOUS) MPI_START(handle) \end{verbatim} {\bf MPI\_RIRECV(buf, len, source, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_RECV(source, tag, context) MPI_BUFFER(handle, CONTIGUOUS, (len)) MPI_START_MODE(handle, READY) MPI_COMPLETE_MODE(handle, REGULAR) MPI_START(handle) \end{verbatim} {\bf MPI\_RIRECVS(buf, numblk, lenblk, stride, len, source, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_RECV(source, tag, context) MPI_BUFFER(handle, STRIDED, buf, (numblk, lenblk, stride)) MPI_START_MODE(handle, READY) MPI_COMPLETE_MODE(handle, REGULAR) MPI_START(handle) \end{verbatim} {\bf MPI\_RIRECVG(vector, len, source, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_RECV(source, tag, context) MPI_BUFFER(handle, GENERAL, (vector)) MPI_START_MODE(handle, READY) MPI_COMPLETE_MODE(handle, REGULAR) MPI_START(handle) \end{verbatim} {\bf MPI\_RSIRECV(buf, len, source, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_RECV(source, tag, context) MPI_BUFFER(handle, CONTIGUOUS, (len)) MPI_START_MODE(handle, READY) MPI_COMPLETE_MODE(handle, SYNCHRONOUS) MPI_START(handle) \end{verbatim} {\bf MPI\_RSIRECVS(buf, numblk, lenblk, stride, len, source, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_RECV(source, tag, context) MPI_BUFFER(handle, STRIDED, buf, (numblk, lenblk, stride)) MPI_START_MODE(handle, READY) MPI_COMPLETE_MODE(handle, SYNCHRONOUS) MPI_START(handle) \end{verbatim} {\bf MPI\_RSIRECVG(vector, len, source, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_RECV(source, tag, context) MPI_BUFFER(handle, GENERAL, (vector)) MPI_START_MODE(handle, READY) MPI_COMPLETE_MODE(handle, SYNCHRONOUS) MPI_START(handle) \end{verbatim} \subsubsection{Completion} A communication operation can be completed in two ways: (i) By a call to {\tt MPI\_WAIT}. Such call returns when the corresponding operation has completed. (ii) by a call to {\tt MPI\_STATUS}. Such call returns successfully if the communication operation has completed; it may also return unsuccessfully if the operation has not completed. See Section~\ref{subsec:complete_ops} for a discussion of the two completion operation types. {\bf MPI\_WAIT\_SEND(handle)} \\ is \begin{verbatim} MPI_COMPLETE_SEND(handle) MPI_FREE(handle) \end{verbatim} {\bf MPI\_WAIT\_RECV(handle, len, source, type)} \\ is \begin{verbatim} MPI_COMPLETE_RECV(handle, len, source, type) MPI_FREE(handle) \end{verbatim} {\bf MPI\_STATUS\_SEND(handle, flag)} \\ is \begin{verbatim} MPI_CHECK_SEND(handle, flag) if (flag) MPI_FREE(handle) \end{verbatim} {\bf MPI\_STATUS\_RECV(handle, len, source, type)} \\ is \begin{verbatim} MPI_CHECK_RECV(handle, flag, len, source, type) if (flag) MPI_FREE(handle) \end{verbatim} \paragraph*{Multiple Receives} It is convenient to be able to wait for any successful receive in a set, rather than having to wait for specific message. {\bf MPI\_WAITANY\_RECV ( list\_of\_handles, index, len, source, tag)} \\ {\bf list\_of\_handles} is an array containing $(\rm n, handle_1 , ... , handle_n)$. Blocks until one of the operations associated with handles in the array has completed. Returns the index of that handle in the array, and the length, source and tag of the received message. {\bf MPI\_WAITANY\_RECV ( list\_of\_handles, index, len, source, tag} \\ is \begin{verbatim} {MPI_WAIT_RECV (handle[1], len, source, tag); index = 1} || ... || {MPI_WAIT_RECV (handle[n], len, source, tag); index = n} \end{verbatim} (``$||$'' indicates alternation; one of the alternatives is chosen, nondeterministically.) \discuss{ An alternative definition of {\tt WAITANY} is to require that the first available message in the list is received. I.e. {\bf MPI\_WAITANY\_RECV ( list\_of\_handles, index, len, source, tag)} \\ is \\ {\tt for(index=1; i$<=$n; i++) \{ \\ \ \ \ MPI\_STATUS\_RECV (handle[index], flag, len, source, tag); \\ \ \ \ if(flag) break; \\ \ \ \ \} } A difference would occur in the case where two receives in the list match the same message; in the original definition any of the two may succeed, whereas in the amended definition only the first one can succeed. This additional determinism, may, on the other hand, restrict concurrency in the implementation of {\tt WAITANY}. It would be nice to have a WAITALL that waits for the completion of all operations listed in the list\_of\_handles. The most useful option might be, in fact, ``wait for the completion of ALL pending operations in the current context'', which could be specified by a ``wildcard'' list\_of\_handles (e.g., a length zero list). } \paragraph*{Probe} It is convenient for a receiver to be able to check for pending messages that can be received, before executing a receive operation. This, for example, would allow the receiver to allocate a receive buffer of the right size, when the size of incoming messages is not known ahead of time. The {\bf PROBE} call specifies an envelope pattern. It returns a handle to an available message with an envelope that matches the pattern, and that does not match any pending receive. It also returns the actual values of {\tt len} {\tt source} and {\tt tag}. The message is now ``locked'' and can be received only by an operation that uses the handle returned. If there is no matching message, then {\tt probe} returns with a NULL handle. {\bf MPI\_PROBE(len, source, tag, context, handle)} \\ If there is a matching message that is not matched by a previously posted nonblocking receive, returns message length, the actual values of source and tag, and a handle that must be used to receive the message. Otherwise, returns a handle with value {\bf NULL} (NULL is a named constant). {\bf MPI\_PRECV ( buf, len, handle)} \\ Nonblocking receive of probed message into a contiguous buffer. {\bf MPI\_PRECVS ( buf, numblk, lenblk, stride, handle)} \\ Nonblocking receive of probed message into a fixed stride buffer. {\bf MPI\_PRECVG ( datavector, handle)} \\ Nonblocking receive of probed message into a buffer specified by a data vector. \discuss{ May want a specific value for NULL, e.g., NULL=0 May need to return more than length, for typed messages. } \subsection{Correctness} \discuss{The material in this section has not yet been discussed by MPIF. Some or all of it is likely to move to Section~\ref{sec:formal}. It is incorporated here for completeness.} \subsubsection{Order} MPI preserves the order of messages between any fixed pair of processes. In other words, if process A executes two successive send {\tt start} suboperations, process B executes two successive receive {\tt start} operations, and both receives match either sends, then the first receive will receive the message sent by the first send, and the second receive will receive the message sent by the second send. The last paragraph assumes that the send {\tt start} operations are ordered by the program order at process A, and the receive {\tt start} operations are ordered by the program order at process B. If a process is multithreaded and the operations are executed by distinct threads, then the semantics of the threaded system may not define an order between the two operations, in which case the condition is void. \subsubsection{Progress and Fairness} We can model the execution of MPI programs as an interaction between executing processes that execute each their own program, and the {\bf communication subsystem}. The communication subsystem may have various constraints on the amount of resources it can use. E.g.: Bounds on the number and total sizes of active handles. Such bound can be global, per node, or per pair of communicating nodes. Bounds on the number and total sizes of messages buffered in the system. Such bound can, again, be global, per node, or per pair of communicating node. In addition, a message may be buffered at the sender, at the receiver, at both, or perhaps at another place altogether. Thus, it will be difficult to set rules on resource management of the communication subsystem. However, it is generally expected that implementers will provide information on the mechanism used for resource allocation, and that query and set functions will allow to query and possibly control the amount of available resources. We provide in this section a set of minimal requirements on the communication subsystem. Programs that execute on any subsystem that fulfils these minimal requirements are {\bf safe} and will port to any MPI implementation. {\bf Unsafe} programs may execute on some MPI implementations, depending on the amount of available resources and the implementation used for the MPI communication subsystem. Finally {\bf erroneous} programs never execute correctly. (While it is desirable to detect erroneous programs, it is not possible to do so at compile time, and often prohibitive to do so a run time. Thus, the document does not specify a behavior for erroneous programs, although the desired behavior is to return a useful error message.) \begin{enumerate} \item Each process can create at least one communication handle. I.e., if a process executes an {\tt MPI\_INIT\_SEND} or {\tt MPI\_INIT\_RECV} operation, and has no other active handle, then the operation eventually succeeds. It is highly desirable to have generous bounds on the number of concurrently active communication handles each process may have, so that, in practice, {\tt INIT} operations will always be guaranteed to succeed. \item Each process can initiate a communication operation for each active handle. I.e. correct {\tt START} operations always succeed (eventually). \item A send operation is {\bf enabled} if the sending process has issued a {\tt COMPLETE\_SEND} operation and the receiving process has issued a {\tt START} operation for a matching receive. Symmetrically, a receive operation is {\bf enabled} if the receiving process has issued a {\tt COMPLETE\_RECV} operation and the sending process has issued a {\tt START} operation for a matching send. An enabled operation may become {\bf disabled} either because it completes successfully or, in the case of a receive, because the matching message is successfully received by another receive operation. {\bf An enabled operation either completes successfully or becomes permanently disabled.} \item A {\tt FREE} operation always succeeds (eventually). \end{enumerate} The four conditions guarantee progress in the communication subsystem. The third condition guarantee (weak) fairness among competing communication requests. Examples (involving two processors with ids 1 and 2) The following program is safe, and should always succeed. \begin{verbatim} IF (GETID() == 1) { MPI_SEND(buf=sendbuf, len=1000, dest=2, tag=0); MPI_RECV(buf=recvbuf, len=1000, source=2, tag=0); } ELSE \* (GETID() == 2) *\ { MPI_RECV(buf=recvbuf, len=1000, source=1, tag=0); MPI_SEND(buf=sendbuf, len=1000, dest=1, type=0); } \end{verbatim} The following program is erroneous, and should always fail. \begin{verbatim} IF (GETID() == 1) { MPI_SSEND(buf=sendbuf, len=1000, dest=2, tag=0); MPI_SRECV(buf=recvbuf, len=1000, source=2, tag=0); } ELSE \* (GETID() == 2) *\ { MPI_SSEND(buf=sendbuf, len=1000, dest=1, tag=0); MPI_SRECV(buf=recvbuf, len=1000, source=1, tag=0); } \end{verbatim} The send operation of the 1st process can complete only if the matching receive of the second processor is executed; the send operation of the second processor can complete only if the matching receive of the first processor is executed. This program will deadlock. The following program is unsafe, and may succeed or fail, depending on implementation. \begin{verbatim} IF (GETID() == 1) THEN { MPI_SEND(buf=sendbuf, len=1000000, dest=2, tag=0); MPI_RECV(buf=recvbuf, len=1000000, source=2, tag=0); } ELSE \* (GETID() == 2) *\ { MPI_SEND(buf=sendbuf, len=1000000, dest=1, tag=0); MPI_RECV(buf=recvbuf, len=1000000, source=1, tag=0); } \end{verbatim} The message sent by each process has to be copied out before the send operation returns and the receive operation starts. For the program to complete, it is necessary that at least one of the two messages sent is buffered out of either processes' address space. Thus, this program can succeed only if the communication system has sufficient buffer space to buffer 1 MgB of data. If additional requirements will become part of the standard (e.g., bounds on the minimal number of concurrently active handles that need be supported, then further programs become safe. \subsection{Error Handling} \subsubsection{Communication Errors} It is assumed that MPI is implemented on top of an error-free communication subsystem: A message sent is always received correctly, and the user does not need to check for transmission errors, time-outs, and the likes. In other words, MPI does not provide mechanisms to deal with failures in the underlying communication subsystem -- it the responsibility of the MPI implementer to insulate the user from such errors (or to reflect them as global program failures). The same holds true for node failures. The other errors can be divided in two classes: \begin{description} \item[resource errors] Errors that occur in an unsafe programs because of limitations on the resources of the communication subsystem, but would not occur if these subsystem had unbounded resources (buffer overflow, no available handles, etc.) \item[program errors] Errors that result in an erroneous program that will always fail. (send with no matching receive, send to nonexistent destination, etc.) \end{description} Program errors always result in an exception, if detected. MPI will provide two mechanisms for handling resource errors: In {\em system mode} each MPI call may return with a return code that indicates that a resource error was detected; no exception occur, and the program may recover in normal execution mode. In {\em user mode} a detected resource error causes an exception. \discuss{ Program errors cannot always be distinguished from resource errors. Have to decide how to handle doubtful cases in system mode. Need to change the syntax so that all return values (error code excepted) are returned via parameters. It's reasonable to restrict system mode to C and C++ bindings. } \end{document} From owner-mpi-pt2pt@CS.UTK.EDU Mon Feb 8 09:37:46 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA27880; Mon, 8 Feb 93 09:37:46 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA23139; Mon, 8 Feb 93 09:37:24 -0500 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Mon, 8 Feb 1993 09:37:22 EST Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from watson.ibm.com by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA23131; Mon, 8 Feb 93 09:37:18 -0500 Message-Id: <9302081437.AA23131@CS.UTK.EDU> Received: from YKTVMV by watson.ibm.com (IBM VM SMTP V2R2) with BSMTP id 9955; Mon, 08 Feb 93 09:37:17 EST Date: Mon, 8 Feb 93 09:37:16 EST From: "Marc Snir" To: MPI-PT2PT@CS.UTK.EDU Some (all?) of you may have received a truncated version of this doc at my previous attempt. Trying again. \documentstyle[12pt]{article} \newcommand{\discuss}[1]{ \ \\ \ \\ {\small {\bf Discussion:} #1} \\ \ \\ } \newcommand{\missing}[1]{ \ \\ \ \\ {\small {\bf Missing:} #1} \\ \ \\ } \begin{document} \title{ Point to Point Communication } \author{Marc Snir} \maketitle \section{Point to Point Communication} \subsection{Introduction} This section is a draft of the current proposal for point-to-point communication. We have not discussed in our last meeting the choice of a syntax for the various communication operations. In particular, there have been no discussion on the choice of function names. There has been no decision whether its preferable to use distinct function names or use additional parameters to distinguish between different variants of the same operation. There has been no discussion whether parameters should be kept separated or bundled in one ``capsule'' that is passed as one parameter. There has been no discussion whether we prefer, in Fortran, functions or subroutines. Thus, the choice of a particular syntax in this draft is merely for the sake of exposition, and is subject to change. I have tried to indicate, wherever appropriate, gaps and unresolved issues, using small type. The current draft borrows heavily on the proposal of Gropp and Lusk for a Multi-Level Message Passing Interface and on my previous outline. \subsection{Messages} A message consists of an {\em envelope} and {\em data}. \subsubsection{Data} \paragraph*{Untyped Data} The simplest (and most used) message data is untyped. Such data consists of a sequence of bytes. Untyped messages can be used to transfer arbitrary typed values between processors with identical data representations; thus, untyped messages are sufficient in a homogeneous environment, where no type conversion is required. \discuss{ Is it legitimate to require that message length be a multiple of 4 or 8? -- or should we require that any number of bytes be OK, with communication perhaps faster for multiples of 4 or 8?} \paragraph*{Typed data} \missing{Need to add here typed messages.} \subsubsection{Envelope} \label{subsec:envelope} The following information is associated with each message: \begin{description} \item[source] The id of the sending task \item[dest] The id of the receiving task \item[tag] User defined \item[context] \end{description} All these fields are of type integer. The range of valid values for the {\bf sender} and {\bf receiver} fields is {\tt 0 ... number\_of\_tasks-1}, where {\tt number\_of\_tasks} is the number of tasks currently participating in the computation (i.e., both fields must include a valid task number). The ranges of valid values for {\tt tag} and {\tt context} are implementation dependent, and can be found by calling a suitable query function, as described in Section~\ref{sec:inquiry}. \discuss{ Users would prefer to be able to use any integer as {\tt tag} or {\tt context}. Vendors may prefer small ranges (e.g. 8 or 16 bits) so as to use fast tag and/or context matching hardware. A defined arithmetic type could be used in C for {\tt tag} and {\tt context}. } The {\tt tag} field can be arbitrarily set by the application, and can be used to distinguish different messages. The {\tt context} field is used to distinguish messages produced by different applications or different libraries. Consider the case where an application calls in parallel on all executing nodes a library function. One needs to make sure that a message generated by the library code will not be consumed by the application and vice-versa. To do so, one can \begin{enumerate} \item Synchronize all nodes before and after the call, or \item Follow a programming convention that prevents such confusion (e.g., make sure that the library code generate messages with tags that are distinct from the tags used by the application, and accepts only messages with correct tags). \end{enumerate} The first solution may lead to less efficient code (it prevents a ``loosely synchronous'' library call). The second solution prevents modular code development. The use of a {\tt context} field, together with the context setting mechanisms described in Section~\ref{sec:context}, solve this problem. Finally, we would like to note that the actual mechanism used to associate data with envelope is implementation dependent; some of the information (e.g., {\bf sender} or {\bf receiver}) may be implicit in some context, and may not be explicitly carried by a message. \subsection{Data Buffers} \label{subsec:buffers} The basic point to point communication operations are {\bf send} and {\bf receive}. A {\bf send} operation creates a message; the message data is assembled from the {\bf send buffer}. A {\bf receive} operation consumes a message; the message data is moved into the {\bf receive buffer}. The specification of send or receive buffers uses the same syntax. \paragraph*{Untyped Messages} There are three kinds of buffers for untyped messages. \begin{description} \item[contiguous buffer] A sequence of contiguous bytes in memory, specified by \begin{description} \item[buf] Initial address \item[len] Number of bytes \end{description} \item[constant stride buffer] A sequence of blocks equally spaced and equally sized blocks, specified by \begin{description} \item[buf] Initial address \item[numblk] Number of blocks \item[lenblk] Number of bytes in each block \item[stride] Number of bytes between the start of each block \end{description} Note that a constant stride buffer becomes a contiguous buffer when {\tt stride = lenblk}. \item[general scatter gather] A sequence of constant stride buffers, specified by a {\bf data\_vector} of the form {\tt (number\_of\_items, buf1, numblk1, lenblk1, stride1, ... bufn, numblkn, lenblkn, striden)}. \end{description} \discuss{ Since Fortran 77 does not have a pointer type, an MPI library that is strictly Fortran 77 compliant would need a different send routine for each possible type of the variable {\tt buf}. This is not very practical. I suggest to ignore the issue and rely on the fact that Fortran 77 compilers and linkers are very unlikely to do type checking across separately compiled modules and are passing routine parameters by reference. The C binding can use pointers, and the Fortran 90 binding can use overloaded subroutines (for basic types). Data vectors are more of a problem, since we now need pointers as components of the data vector. The Posix Fortran 77 Language Interface IEEE standard does not provide a general enough mechanism for that purpose. We might restrict the use of data vectors to C. (Of course, sophisticated programmers would be able to call C routines from Fortran, and nonstandard solutions might also be provided.) Or, we might require that all buffer addresses are integer displacements with respect to the first address, in which case all entries are integer. Is it acceptable to have a (small) fixed upper bound on the length of data vectors? } \paragraph*{Typed Buffers} \missing{Data buffers for typed messages.} \subsection{Receive Criteria} The selection of a message by a receive operation is done according to the value of the message envelope. The receive operation specifies an {\bf envelope pattern}; a message can be received by that receive operation only if its envelope matches that pattern. A pattern specifies values for the {\tt source}, {\tt tag} and {\tt context} fields of the message envelope. In addition, the value for the {\tt dest} field is set, implicitly, to be equal to the receiving process id. The receiver may specify a {\bf DONTCARE} value for {\tt source}, or {\tt tag}, indicating that any source and/or tag are acceptable. It cannot specify a DONTCARE value for {\tt context} or {\tt dest}. Thus, a message can be received by a receive operation only if it is addressed to the receiving task, has a matching context, has matching source unless source=DONTCARE in the pattern, and has a matching tag unless tag=DONTCARE in the pattern. The length of the received message must be less or equal the length of the receive buffer. I.e., all incoming data must fit, without truncation, into the receive buffer. It is erroneous to receive a message which length exceed the receive buffer, and the outcome of program where this occurs is undetermined. \discuss{ The DONTCARE value can be a fixed value (e.g. -1), and/or a named constant. It might be useful to have the option to require exact size match of incoming message and receive buffer. Some would also like to see truncation supported. } \subsection{Communication Handles} One can consider (most) communication operations to consist of the following suboperations: \begin{description} \item[INIT(operation, params, handle)] Process provides all relevant parameters for its participation in the communication operation (type of operation, data buffer, tag, participants, etc.). A handle is created that identifies the operation. \item[START(handle)] The communication operation is started \item[COMPLETE(handle)] The communication operation is completed. \item[FREE(handle)] The handle, and associated resources are freed. \end{description} Correct invocation of these suboperations is a sequence of the form \[ \bf INIT \ (START \ COMPLETE )^* \ FREE .\] I.e., a handle needs be created before communication occurs; it can be reused only after the previous use has completed; and it needs to be freed eventually (of course, one can assume that all handles are freed at program termination, by default). A user may directly invokes these suboperations. This would allow to amortize the overhead of setting up a communication over many successive uses of the same handle, and allows to overlap communication and computation. Simpler communication operations combine several of these suboperations into one operation, thus simplifying the use of communication primitives. Thus, one only needs to specify precisely the semantics of these suboperations in order to specify the semantics of MPI point to point communication operations. We say that a communication operation (send or receive) is {\bf posted} once a {\bf start} suboperation was invoked; the operation is {\bf completed} once the {\bf complete} suboperation completes. A send and a receive operation {\bf match} if the receive pattern specified by the receive matches the message envelope created by the send. \subsubsection{Handle Creation} A handle for a send operation is created by a call to {\bf MPI\_INIT\_SEND}, followed by a set of calls that set the various parameters associated with the handle. The handle cannot be used until all parameters have been set. A call to {\bf MPI\_INIT\_RECV} is similarly used for creating a handle for a receive operation. Handle creation is a local operation that need not involve communication with a remote process. {\bf handle = MPI\_INIT\_SEND (dest, tag, context)} \\ Create a send handle. All three parameters must be specified. {\bf handle = MPI\_INIT\_RECV (source, tag, context)} \\ Create a send handle. DONTCARE values may be used for the first two parameters. See Section~\ref{subsec:envelope} for a discussion of source, tag and context. \discuss{ An alternative design, proposed by Gropp and Lusk, is to have each newly created handle associated with default parameters that can be later modified. In their proposal, too, all modifications must occur before the first use.} \subsubsection{Buffer specification} The {\bf MPI\_BUFFER} function associates a buffer with a handle. {\bf MPI\_BUFFER(handle, buftype, params)}\\ \begin{tabbing} {\bf buftype} \ \ \ \ \ \ \ \ \ \ \ \= {\bf params} \\ CONTIGUOUS \> buf, len \\ STRIDED \> buf, numblk, lenblk, stride \\ GENERAL \> data\_vector \end{tabbing} See Section~\ref{subsec:buffers} for a description of these buffer types and parameters. The {\tt buftype} parameter is an integer, with named constants provided for the three values. The {\tt params} parameter list consists of an address (pointer) for {\tt buf}, and a one-dimensional array of integers to specify the remaining values. See Section~\ref{subsec:buffers} for a discussion of data vectors. \missing{typed buffers} \discuss{ Some may want specific values for {\tt buftype}, e.g. CONTIGUOUS =0, STRIDED=1, GENERAL=2. } \subsubsection{Start Mode} The {\bf MPI\_START\_MODE} function selects a mode for starting a communication operation. {\bf MPI\_START\_MODE(mode)} The mode parameter is an integer, with named constants provided for its two values. There are two modes: \begin{description} \item[REGULAR] The operation may start whether or not a matching communication operation has been posted. \item[READY] The operation my start only if a matching communication operation has been posted. \end{description} Thus, a {\bf ready send} can start only if a matching receive is already posted; otherwise the operation is erroneous and its outcome is undefined. A {\bf ready receive} can start only if a matching send is already posted; otherwise the operation is erroneous and its outcome is undefined. \discuss{ May want to fix the values, e.g., REGULAR=0 and READY=1. We discussed only ready send, not ready receive. Ready send is useful when message passing is implemented in a ``push'' style, where the sender moves the message to the receiver (common case). Ready receive would be useful when message passing is implemented in a ``pull'' style, where the receiver moves the data from the sender. } \subsubsection{Completion Mode} \label{subsec:completion} The {\bf MPI\_COMPLETE\_MODE} function selects a mode for completing a communication operation. {\bf MPI\_COMPLETE\_MODE(mode)} The mode parameter is an integer, with named constants provided for its two values. There are two modes: \begin{description} \item[REGULAR] The operation completes locally, irrespective of the completion of the matching operation. \item[SYNCHRONOUS] The operation completes only if the matching operation completes too. \end{description} A regular send can complete as soon as the message has been composed: data has been copied out of the send buffer and the values used for the envelope have been stored. The sender may now update the send buffer. A regular send can complete before the matching receive has completed. A regular receive can complete as soon as the message has been consumed: data is available in the receive buffer and all values returned by the send complete operation are available. The receiver may now access the receive buffer. A regular receive may complete before the matching send has completed (It may seem strange, at first blush, that a receive may complete before the matching send completes. However, one should keep in mind that an operation completes only when the {\tt complete} suboperation terminates. Thus, even though data has been copied out and the message was sent, a send has not not yet completed if the {\tt send complete} suboperation has not yet been invoked. Informally, the send (receive) completes only when the sender (receiver) knows the operation has completed.) A synchronous communication operation completes only if the matching operation completes too. Thus, a synchronous send can complete as soon as the all operations preceding the completion of the matching receive have terminated. A synchronous receive may terminate as soon as the message has been consumed and all operations preceding the completion of the matching send have terminated. Informally, the completion of the matching send and receive operations are synchronized. Matching communication operations must have the same completion mode. I.e., either both send and receive are regular or both are synchronous. A program where a regular receive matches a synchronous send (or vice versa) is erroneous and its outcome is undefined. \discuss{ May want to fix the values, e.g., REGULAR=0 and SYNCHRONOUS=1. } \subsubsection{Communication Start} {\bf MPI\_START(handle)} \\ The {\tt MPI\_START} function starts the execution of a communication operation (send or receive). A sender should not update the send buffer after a send operation has started and until it is completed. A receiver should not access the receive buffer after a receive operation was started and until it is completed. A program that does not satisfy this condition is erroneous and its outcome is undetermined. \discuss{ Some might prefer two distinct functions: {\bf MPI\_SEND\_START} and {\bf MPI\_RECV\_START}. } \subsubsection{Communication Completion} \label{subsec:complete_ops} {\bf MPI\_COMPLETE\_SEND ( handle)} \\ A call to {\bf MPI\_COMPLETE\_SEND} returns when the send operation identified by {\tt handle} is complete (see Section~\ref{subsec:completion} for a description of completion criteria). {\bf MPI\_COMPLETE\_RECV ( handle, len, source, tag)} \\ A call to {\bf MPI\_COMPLETE\_RECV} returns when the receive operation identified by {\tt handle} is complete (see Section~\ref{subsec:completion} for a description of completion criteria). The call returns the number of bytes received, and the source and tag of the message received. {\bf MPI\_CHECK\_SEND ( handle, flag)} \\ A call to {\bf MPI\_CHECK\_SEND} returns {\tt flag=true} if the send operation identified by {\tt handle} is complete (see Section~\ref{subsec:completion} for a description of completion criteria). It returns {\tt flag=false}, otherwise. A successful return of {\bf MPI\_CHECK\_SEND} has the same semantics as a return of {\bf MPI\_COMPLETE\_SEND}. {\bf MPI\_CHECK\_RECV ( handle, flag, len, source, tag)} \\ A call to {\bf MPI\_CHECK\_RECV} returns {\tt flag=true} if the receive operation identified by {\tt handle} is complete (see Section~\ref{subsec:completion} for a description of completion criteria). In such case the call returns the number of bytes received, and the source and tag of the message received. The call returns {\tt flag=false}, otherwise. In such case, the return values of {\tt len}, {\tt source} and {\tt tag} are undefined. Implementation notes: A call to {\tt MPI\_COMPLETE} blocks only the executing thread. If the executing process is multithreaded, then other threads within the process can be scheduled for execution. A call to {\tt MPI\_COMPLETE\_SEND(handle)} has the same semantics as a busy loop of the form \begin{verbatim} repeat MPI_CHECK_SEND(handle, flag) until(flag=true) \end{verbatim} and similarly for receives. The use of a blocking receive operation ({\tt MPI\_COMPLETE}) allows the operating system to deschedule the blocked thread and schedule another thread for execution, if such is available. The use of a nonblocking receive operation ({\tt MPI\_CHECK}) allows the user to schedule alternative activities within a single thread of execution. The intended implementation of {\tt MPI\_CHECK} is for that operation to return as soon as possible. Note that it is correct, but inefficient, to implement {\tt MPI\_CHECK} via a call to {\tt MPI\_COMPLETE}, in which case, {\tt MPI\_CHECK} always returns {\tt true}. \discuss{ One might use {\tt len=-1} to indicate unsuccessful return of {\tt MPI\_CHECK\_RECV} } \subsection{Blocking Communication} Blocking send and receive operations combine all four suboperations into one call. The operation returns only when the communication completes and no communication handle persists after the call completed. We use the following naming convention for such operations: \[ \left[ \begin{array}{c} - \\ \bf r \end{array} \right] \left[ \begin{array}{c} - \\ \bf s \end{array} \right] \left[ \begin{array}{c} \bf send \\ \bf recv \end{array} \right] \left[ \begin{array}{c} - \\ \bf s \\ \bf g \end{array} \right] \] The first letter (void or {\bf r}) indicates the start mode (regular or ready). The second letter (void or {\bf s}) indicates the completion mode (regular or synchronous). The last letter (void, {\bf s} or {\bf g}) indicates the buffer type (contiguous, strided or general). The corresponding 12 send and 12 receive operations are listed below. {\bf MPI\_SEND(buf, len, dest, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_SEND(dest, tag, context) MPI_BUFFER(handle, CONTIGUOUS, (len)) MPI_START_MODE(handle, REGULAR) MPI_COMPLETE_MODE(handle, REGULAR) MPI_START(handle) MPI_COMPLETE_SEND(handle) MPI_FREE(handle) \end{verbatim} {\bf MPI\_SENDS(buf, numblk, lenblk, stride, dest, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_SEND(dest, tag, context) MPI_BUFFER(handle, STRIDED, buf, (numblk, lenblk, stride)) MPI_START_MODE(handle, REGULAR) MPI_COMPLETE_MODE(handle, REGULAR) MPI_START(handle) MPI_COMPLETE_SEND(handle) MPI_FREE(handle) \end{verbatim} {\bf MPI\_SENDG(vector, dest, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_SEND(dest, tag, context) MPI_BUFFER(handle, GENERAL, (vector)) MPI_START_MODE(handle, REGULAR) MPI_COMPLETE_MODE(handle, REGULAR) MPI_START(handle) MPI_COMPLETE_SEND(handle) MPI_FREE(handle) \end{verbatim} {\bf MPI\_SSEND(buf, len, dest, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_SEND(dest, tag, context) MPI_BUFFER(handle, CONTIGUOUS, (len)) MPI_START_MODE(handle, REGULAR) MPI_COMPLETE_MODE(handle, SYNCHRONOUS) MPI_START(handle) MPI_COMPLETE_SEND(handle) MPI_FREE(handle) \end{verbatim} {\bf MPI\_SSENDS(buf, numblk, lenblk, stride, dest, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_SEND(dest, tag, context) MPI_BUFFER(handle, STRIDED, buf, (numblk, lenblk, stride)) MPI_START_MODE(handle, REGULAR) MPI_COMPLETE_MODE(handle, SYNCHRONOUS) MPI_START(handle) MPI_COMPLETE_SEND(handle) MPI_FREE(handle) \end{verbatim} {\bf MPI\_SSENDG(vector, dest, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_SEND(dest, tag, context) MPI_BUFFER(handle, GENERAL, (vector)) MPI_START_MODE(handle, REGULAR) MPI_COMPLETE_MODE(handle, SYNCHRONOUS) MPI_START(handle) MPI_COMPLETE_SEND(handle) MPI_FREE(handle) \end{verbatim} {\bf MPI\_RSEND(buf, len, dest, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_SEND(dest, tag, context) MPI_BUFFER(handle, CONTIGUOUS, (len)) MPI_START_MODE(handle, READY) MPI_COMPLETE_MODE(handle, REGULAR) MPI_START(handle) MPI_COMPLETE_SEND(handle) MPI_FREE(handle) \end{verbatim} {\bf MPI\_RSENDS(buf, numblk, lenblk, stride, dest, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_SEND(dest, tag, context) MPI_BUFFER(handle, STRIDED, buf, (numblk, lenblk, stride)) MPI_START_MODE(handle, READY) MPI_COMPLETE_MODE(handle, REGULAR) MPI_START(handle) MPI_COMPLETE_SEND(handle) MPI_FREE(handle) \end{verbatim} {\bf MPI\_RSENDG(vector, dest, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_SEND(dest, tag, context) MPI_BUFFER(handle, GENERAL, (vector)) MPI_START_MODE(handle, READY) MPI_COMPLETE_MODE(handle, REGULAR) MPI_START(handle) MPI_COMPLETE_SEND(handle) MPI_FREE(handle) \end{verbatim} {\bf MPI\_RSSEND(buf, len, dest, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_SEND(dest, tag, context) MPI_BUFFER(handle, CONTIGUOUS, (len)) MPI_START_MODE(handle, READY) MPI_COMPLETE_MODE(handle, SYNCHRONOUS) MPI_START(handle) MPI_COMPLETE_SEND(handle) MPI_FREE(handle) \end{verbatim} {\bf MPI\_RSSENDS(buf, numblk, lenblk, stride, dest, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_SEND(dest, tag, context) MPI_BUFFER(handle, STRIDED, buf, (numblk, lenblk, stride)) MPI_START_MODE(handle, READY) MPI_COMPLETE_MODE(handle, SYNCHRONOUS) MPI_START(handle) MPI_COMPLETE_SEND(handle) MPI_FREE(handle) \end{verbatim} {\bf MPI\_RSSENDG(vector, dest, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_SEND(dest, tag, context) MPI_BUFFER(handle, GENERAL, (vector)) MPI_START_MODE(handle, READY) MPI_COMPLETE_MODE(handle, SYNCHRONOUS) MPI_START(handle) MPI_COMPLETE_SEND(handle) MPI_FREE(handle) \end{verbatim} {\bf MPI\_RECV(buf, len, source, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_RECV(source, tag, context) MPI_BUFFER(handle, CONTIGUOUS, (len)) MPI_START_MODE(handle, REGULAR) MPI_COMPLETE_MODE(handle, REGULAR) MPI_START(handle) MPI_COMPLETE_SEND(handle,len,source,tag) MPI_FREE(handle) \end{verbatim} {\bf MPI\_RECVS(buf, numblk, lenblk, stride, len, source, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_RECV(source, tag, context) MPI_BUFFER(handle, STRIDED, buf, (numblk, lenblk, stride)) MPI_START_MODE(handle, REGULAR) MPI_COMPLETE_MODE(handle, REGULAR) MPI_START(handle) MPI_COMPLETE_RECV(handle, len, source, tag) MPI_FREE(handle) \end{verbatim} {\bf MPI\_RECVG(vector, len, source, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_RECV(source, tag, context) MPI_BUFFER(handle, GENERAL, (vector)) MPI_START_MODE(handle, REGULAR) MPI_COMPLETE_MODE(handle, REGULAR) MPI_START(handle) MPI_COMPLETE_RECV(handle, len, source, tag) MPI_FREE(handle) \end{verbatim} {\bf MPI\_SRECV(buf, len, source, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_RECV(source, tag, context) MPI_BUFFER(handle, CONTIGUOUS, (len)) MPI_START_MODE(handle, REGULAR) MPI_COMPLETE_MODE(handle, SYNCHRONEOUS) MPI_START(handle) MPI_COMPLETE_RECV(handle, len, source, tag) MPI_FREE(handle) \end{verbatim} {\bf MPI\_SRECVS(buf, numblk, lenblk, stride, len, source, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_RECV(source, tag, context) MPI_BUFFER(handle, STRIDED, buf, (numblk, lenblk, stride)) MPI_START_MODE(handle, REGULAR) MPI_COMPLETE_MODE(handle, SYNCHRONOUS) MPI_START(handle) MPI_COMPLETE_RECV(handle, len, source, tag) MPI_FREE(handle) \end{verbatim} {\bf MPI\_SRECVG(vector, len, source, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_RECV(source, tag, context) MPI_BUFFER(handle, GENERAL, (vector)) MPI_START_MODE(handle, REGULAR) MPI_COMPLETE_MODE(handle, SYNCHRONEOUS) MPI_START(handle) MPI_COMPLETE_RECV(handle, len, source, tag) MPI_FREE(handle) \end{verbatim} {\bf MPI\_RRECV(buf, len, source, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_RECV(source, tag, context) MPI_BUFFER(handle, CONTIGUOUS, (len)) MPI_START_MODE(handle, READY) MPI_COMPLETE_MODE(handle, REGULAR) MPI_START(handle) MPI_COMPLETE_RECV(handle, len, source, tag) MPI_FREE(handle) \end{verbatim} {\bf MPI\_RRECVS(buf, numblk, lenblk, stride, len, source, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_RECV(source, tag, context) MPI_BUFFER(handle, STRIDED, buf, (numblk, lenblk, stride)) MPI_START_MODE(handle, READY) MPI_COMPLETE_MODE(handle, REGULAR) MPI_START(handle) MPI_COMPLETE_RECV(handle) MPI_FREE(handle) \end{verbatim} {\bf MPI\_RRECVG(vector, len, source, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_RECV(source, tag, context) MPI_BUFFER(handle, GENERAL, (vector)) MPI_START_MODE(handle, READY) MPI_COMPLETE_MODE(handle, REGULAR) MPI_START(handle) MPI_COMPLETE_RECV(handle, len, soure, tag) MPI_FREE(handle) \end{verbatim} {\bf MPI\_RSRECV(buf, len, source, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_RECV(source, tag, context) MPI_BUFFER(handle, CONTIGUOUS, (len)) MPI_START_MODE(handle, READY) MPI_COMPLETE_MODE(handle, SYNCHRONOUS) MPI_START(handle) MPI_COMPLETE_RECV(handle, len, source, tag) MPI_FREE(handle) \end{verbatim} {\bf MPI\_RSRECVS(buf, numblk, lenblk, stride, len, source, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_RECV(source, tag, context) MPI_BUFFER(handle, STRIDED, buf, (numblk, lenblk, stride)) MPI_START_MODE(handle, READY) MPI_COMPLETE_MODE(handle, SYNCHRONEOUS) MPI_START(handle) MPI_COMPLETE_RECV(handle, len, source, tag) MPI_FREE(handle) \end{verbatim} {\bf MPI\_RSRECVG(vector, len, source, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_RECV(source, tag, context) MPI_BUFFER(handle, GENERAL, (vector)) MPI_START_MODE(handle, READY) MPI_COMPLETE_MODE(handle, SYNCHRONOUS) MPI_START(handle) MPI_COMPLETE_RECV(handle, len, source, tag) MPI_FREE(handle) \end{verbatim} Note that all receive calls return the actual length, tag and source of the received message. This, the {\tt source} and {\tt tag} parameters are INOUT. The {\tt len} parameter is INOUT for contiguous receive buffers, OUT for strided and general receive buffers. {\bf Implementation note:} While these 24 functions can be implemented via calls to functions that implement suboperations, as described in this subsection, an efficient implementation may optimize away these multiple calls, provided it does not change the behavior of correct programs. \discuss{ Not all 24 functions are equally important (ready-synchronous is somewhat esoteric). We can decide to organize the functins so that the most important are in a ``core subset''. But (i) users will use whatever you provide, and will find uses for what you believed to be useless; (ii) there is a virtue in being systematic -- no need to remember what works and what does not; (iii) what is not important can be implemented using multiple calls to the basic functions. The need for an additional {\tt len} parameter for strided and general receives is somewhat annoying. Suggestions? } \subsection{Nonblocking Communication} Nonblocking send and receive operations combine the first two suboperations ({\tt INIT} and {\tt START}) into one call, and the last two suboperations ({\tt COMPLETE} or {\tt CHECK} and {\tt FREE}) into another suboperation. Thus a nonblocking send or receive consists of two suboperations: One that starts the communication, and one that completes the operation and free any associated resources. No communication handle persists after these two suboperations are executed. \subsubsection{Initiation} We use the following naming convention for the first suboperation of a nonblocking communication: \[ \left[ \begin{array}{c} - \\ \bf r \end{array} \right] \left[ \begin{array}{c} - \\ \bf s \end{array} \right] \left[ \begin{array}{c} \bf isend \\ \bf irecv \end{array} \right] \left[ \begin{array}{c} - \\ \bf s \\ \bf g \end{array} \right] \] The first letter (void or {\bf r}) indicates the start mode (regular or ready). The second letter (void or {\bf s}) indicates the completion mode (regular or synchronous). The last letter (void, {\bf s} or {\bf g}) indicates the buffer type (contiguous, strided or general). The corresponding 12 send and 12 receive operations are listed below. The letter {\bf i} in {\bf isend} and {\bf irecv} indicates that the operation is ``immediate'' and that the call may return before the operation is completed. We thus have 12 immediate send operations and 12 immediate receive operations. {\bf MPI\_ISEND(buf, len, dest, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_ISEND(dest, tag, context) MPI_BUFFER(handle, CONTIGUOUS, (len)) MPI_START_MODE(handle, REGULAR) MPI_COMPLETE_MODE(handle, REGULAR) MPI_START(handle) \end{verbatim} {\bf MPI\_ISENDS(buf, numblk, lenblk, stride, dest, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_SEND(dest, tag, context) MPI_BUFFER(handle, STRIDED, buf, (numblk, lenblk, stride)) MPI_START_MODE(handle, REGULAR) MPI_COMPLETE_MODE(handle, REGULAR) MPI_START(handle) \end{verbatim} {\bf MPI\_ISENDG(vector, dest, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_SEND(dest, tag, context) MPI_BUFFER(handle, GENERAL, (vector)) MPI_START_MODE(handle, REGULAR) MPI_COMPLETE_MODE(handle, REGULAR) MPI_START(handle) \end{verbatim} {\bf MPI\_SISEND(buf, len, dest, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_SEND(dest, tag, context) MPI_BUFFER(handle, CONTIGUOUS, (len)) MPI_START_MODE(handle, REGULAR) MPI_COMPLETE_MODE(handle, SYNCHRONOUS) MPI_START(handle) \end{verbatim} {\bf MPI\_SISENDS(buf, numblk, lenblk, stride, dest, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_SEND(dest, tag, context) MPI_BUFFER(handle, STRIDED, buf, (numblk, lenblk, stride)) MPI_START_MODE(handle, REGULAR) MPI_COMPLETE_MODE(handle, SYNCHRONOUS) MPI_START(handle) \end{verbatim} {\bf MPI\_SISENDG(vector, dest, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_SEND(dest, tag, context) MPI_BUFFER(handle, GENERAL, (vector)) MPI_START_MODE(handle, REGULAR) MPI_COMPLETE_MODE(handle, SYNCHRONEOUS) MPI_START(handle) \end{verbatim} {\bf MPI\_RISEND(buf, len, dest, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_SEND(dest, tag, context) MPI_BUFFER(handle, CONTIGUOUS, (len)) MPI_START_MODE(handle, READY) MPI_COMPLETE_MODE(handle, REGULAR) MPI_START(handle) \end{verbatim} {\bf MPI\_RISENDS(buf, numblk, lenblk, stride, dest, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_SEND(dest, tag, context) MPI_BUFFER(handle, STRIDED, buf, (numblk, lenblk, stride)) MPI_START_MODE(handle, READY) MPI_COMPLETE_MODE(handle, REGULAR) MPI_START(handle) \end{verbatim} {\bf MPI\_RISENDG(vector, dest, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_SEND(dest, tag, context) MPI_BUFFER(handle, GENERAL, (vector)) MPI_START_MODE(handle, READY) MPI_COMPLETE_MODE(handle, REGULAR) MPI_START(handle) \end{verbatim} {\bf MPI\_RSISEND(buf, len, dest, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_SEND(dest, tag, context) MPI_BUFFER(handle, CONTIGUOUS, (len)) MPI_START_MODE(handle, READY) MPI_COMPLETE_MODE(handle, SYNCHRONEOUS) MPI_START(handle) \end{verbatim} {\bf MPI\_RSISENDS(buf, numblk, lenblk, stride, dest, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_SEND(dest, tag, context) MPI_BUFFER(handle, STRIDED, buf, (numblk, lenblk, stride)) MPI_START_MODE(handle, READY) MPI_COMPLETE_MODE(handle, SYNCHRONOUS) MPI_START(handle) \end{verbatim} {\bf MPI\_RSISENDG(vector, dest, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_SEND(dest, tag, context) MPI_BUFFER(handle, GENERAL, (vector)) MPI_START_MODE(handle, READY) MPI_COMPLETE_MODE(handle, SYNCHRONOUS) MPI_START(handle) \end{verbatim} {\bf MPI\_IRECV(buf, len, source, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_RECV(source, tag, context) MPI_BUFFER(handle, CONTIGUOUS, (len)) MPI_START_MODE(handle, REGULAR) MPI_COMPLETE_MODE(handle, REGULAR) MPI_START(handle) \end{verbatim} {\bf MPI\_IRECVS(buf, numblk, lenblk, stride, len, source, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_RECV(source, tag, context) MPI_BUFFER(handle, STRIDED, buf, (numblk, lenblk, stride)) MPI_START_MODE(handle, REGULAR) MPI_COMPLETE_MODE(handle, REGULAR) MPI_START(handle) \end{verbatim} {\bf MPI\_IRECVG(vector, len, source, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_RECV(source, tag, context) MPI_BUFFER(handle, GENERAL, (vector)) MPI_START_MODE(handle, REGULAR) MPI_COMPLETE_MODE(handle, REGULAR) MPI_START(handle) \end{verbatim} {\bf MPI\_SIRECV(buf, len, source, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_RECV(source, tag, context) MPI_BUFFER(handle, CONTIGUOUS, (len)) MPI_START_MODE(handle, REGULAR) MPI_COMPLETE_MODE(handle, SYNCHRONOUS) MPI_START(handle) \end{verbatim} {\bf MPI\_SIRECVS(buf, numblk, lenblk, stride, len, source, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_RECV(source, tag, context) MPI_BUFFER(handle, STRIDED, buf, (numblk, lenblk, stride)) MPI_START_MODE(handle, REGULAR) MPI_COMPLETE_MODE(handle, SYNCHRONOUS) MPI_START(handle) \end{verbatim} {\bf MPI\_SIRECVG(vector, len, source, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_RECV(source, tag, context) MPI_BUFFER(handle, GENERAL, (vector)) MPI_START_MODE(handle, REGULAR) MPI_COMPLETE_MODE(handle, SYNCHRONOUS) MPI_START(handle) \end{verbatim} {\bf MPI\_RIRECV(buf, len, source, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_RECV(source, tag, context) MPI_BUFFER(handle, CONTIGUOUS, (len)) MPI_START_MODE(handle, READY) MPI_COMPLETE_MODE(handle, REGULAR) MPI_START(handle) \end{verbatim} {\bf MPI\_RIRECVS(buf, numblk, lenblk, stride, len, source, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_RECV(source, tag, context) MPI_BUFFER(handle, STRIDED, buf, (numblk, lenblk, stride)) MPI_START_MODE(handle, READY) MPI_COMPLETE_MODE(handle, REGULAR) MPI_START(handle) \end{verbatim} {\bf MPI\_RIRECVG(vector, len, source, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_RECV(source, tag, context) MPI_BUFFER(handle, GENERAL, (vector)) MPI_START_MODE(handle, READY) MPI_COMPLETE_MODE(handle, REGULAR) MPI_START(handle) \end{verbatim} {\bf MPI\_RSIRECV(buf, len, source, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_RECV(source, tag, context) MPI_BUFFER(handle, CONTIGUOUS, (len)) MPI_START_MODE(handle, READY) MPI_COMPLETE_MODE(handle, SYNCHRONOUS) MPI_START(handle) \end{verbatim} {\bf MPI\_RSIRECVS(buf, numblk, lenblk, stride, len, source, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_RECV(source, tag, context) MPI_BUFFER(handle, STRIDED, buf, (numblk, lenblk, stride)) MPI_START_MODE(handle, READY) MPI_COMPLETE_MODE(handle, SYNCHRONOUS) MPI_START(handle) \end{verbatim} {\bf MPI\_RSIRECVG(vector, len, source, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_RECV(source, tag, context) MPI_BUFFER(handle, GENERAL, (vector)) MPI_START_MODE(handle, READY) MPI_COMPLETE_MODE(handle, SYNCHRONOUS) MPI_START(handle) \end{verbatim} \subsubsection{Completion} A communication operation can be completed in two ways: (i) By a call to {\tt MPI\_WAIT}. Such call returns when the corresponding operation has completed. (ii) by a call to {\tt MPI\_STATUS}. Such call returns successfully if the communication operation has completed; it may also return unsuccessfully if the operation has not completed. See Section~\ref{subsec:complete_ops} for a discussion of the two completion operation types. {\bf MPI\_WAIT\_SEND(handle)} \\ is \begin{verbatim} MPI_COMPLETE_SEND(handle) MPI_FREE(handle) \end{verbatim} {\bf MPI\_WAIT\_RECV(handle, len, source, type)} \\ is \begin{verbatim} MPI_COMPLETE_RECV(handle, len, source, type) MPI_FREE(handle) \end{verbatim} {\bf MPI\_STATUS\_SEND(handle, flag)} \\ is \begin{verbatim} MPI_CHECK_SEND(handle, flag) if (flag) MPI_FREE(handle) \end{verbatim} {\bf MPI\_STATUS\_RECV(handle, len, source, type)} \\ is \begin{verbatim} MPI_CHECK_RECV(handle, flag, len, source, type) if (flag) MPI_FREE(handle) \end{verbatim} \paragraph*{Multiple Receives} It is convenient to be able to wait for any successful receive in a set, rather than having to wait for specific message. {\bf MPI\_WAITANY\_RECV ( list\_of\_handles, index, len, source, tag)} \\ {\bf list\_of\_handles} is an array containing $(\rm n, handle_1 , ... , handle_n)$. Blocks until one of the operations associated with handles in the array has completed. Returns the index of that handle in the array, and the length, source and tag of the received message. {\bf MPI\_WAITANY\_RECV ( list\_of\_handles, index, len, source, tag} \\ is \begin{verbatim} {MPI_WAIT_RECV (handle[1], len, source, tag); index = 1} || ... || {MPI_WAIT_RECV (handle[n], len, source, tag); index = n} \end{verbatim} (``$||$'' indicates alternation; one of the alternatives is chosen, nondeterministically.) \discuss{ An alternative definition of {\tt WAITANY} is to require that the first available message in the list is received. I.e. {\bf MPI\_WAITANY\_RECV ( list\_of\_handles, index, len, source, tag)} \\ is \\ {\tt for(index=1; i$<=$n; i++) \{ \\ \ \ \ MPI\_STATUS\_RECV (handle[index], flag, len, source, tag); \\ \ \ \ if(flag) break; \\ \ \ \ \} } A difference would occur in the case where two receives in the list match the same message; in the original definition any of the two may succeed, whereas in the amended definition only the first one can succeed. This additional determinism, may, on the other hand, restrict concurrency in the implementation of {\tt WAITANY}. It would be nice to have a WAITALL that waits for the completion of all operations listed in the list\_of\_handles. The most useful option might be, in fact, ``wait for the completion of ALL pending operations in the current context'', which could be specified by a ``wildcard'' list\_of\_handles (e.g., a length zero list). } \paragraph*{Probe} It is convenient for a receiver to be able to check for pending messages that can be received, before executing a receive operation. This, for example, would allow the receiver to allocate a receive buffer of the right size, when the size of incoming messages is not known ahead of time. The {\bf PROBE} call specifies an envelope pattern. It returns a handle to an available message with an envelope that matches the pattern, and that does not match any pending receive. It also returns the actual values of {\tt len} {\tt source} and {\tt tag}. The message is now ``locked'' and can be received only by an operation that uses the handle returned. If there is no matching message, then {\tt probe} returns with a NULL handle. {\bf MPI\_PROBE(len, source, tag, context, handle)} \\ If there is a matching message that is not matched by a previously posted nonblocking receive, returns message length, the actual values of source and tag, and a handle that must be used to receive the message. Otherwise, returns a handle with value {\bf NULL} (NULL is a named constant). {\bf MPI\_PRECV ( buf, len, handle)} \\ Nonblocking receive of probed message into a contiguous buffer. {\bf MPI\_PRECVS ( buf, numblk, lenblk, stride, handle)} \\ Nonblocking receive of probed message into a fixed stride buffer. {\bf MPI\_PRECVG ( datavector, handle)} \\ Nonblocking receive of probed message into a buffer specified by a data vector. \discuss{ May want a specific value for NULL, e.g., NULL=0 May need to return more than length, for typed messages. } \subsection{Correctness} \discuss{The material in this section has not yet been discussed by MPIF. Some or all of it is likely to move to Section~\ref{sec:formal}. It is incorporated here for completeness.} \subsubsection{Order} MPI preserves the order of messages between any fixed pair of processes. In other words, if process A executes two successive send {\tt start} suboperations, process B executes two successive receive {\tt start} operations, and both receives match either sends, then the first receive will receive the message sent by the first send, and the second receive will receive the message sent by the second send. The last paragraph assumes that the send {\tt start} operations are ordered by the program order at process A, and the receive {\tt start} operations are ordered by the program order at process B. If a process is multithreaded and the operations are executed by distinct threads, then the semantics of the threaded system may not define an order between the two operations, in which case the condition is void. \subsubsection{Progress and Fairness} We can model the execution of MPI programs as an interaction between executing processes that execute each their own program, and the {\bf communication subsystem}. The communication subsystem may have various constraints on the amount of resources it can use. E.g.: Bounds on the number and total sizes of active handles. Such bound can be global, per node, or per pair of communicating nodes. Bounds on the number and total sizes of messages buffered in the system. Such bound can, again, be global, per node, or per pair of communicating node. In addition, a message may be buffered at the sender, at the receiver, at both, or perhaps at another place altogether. Thus, it will be difficult to set rules on resource management of the communication subsystem. However, it is generally expected that implementers will provide information on the mechanism used for resource allocation, and that query and set functions will allow to query and possibly control the amount of available resources. We provide in this section a set of minimal requirements on the communication subsystem. Programs that execute on any subsystem that fulfils these minimal requirements are {\bf safe} and will port to any MPI implementation. {\bf Unsafe} programs may execute on some MPI implementations, depending on the amount of available resources and the implementation used for the MPI communication subsystem. Finally {\bf erroneous} programs never execute correctly. (While it is desirable to detect erroneous programs, it is not possible to do so at compile time, and often prohibitive to do so a run time. Thus, the document does not specify a behavior for erroneous programs, although the desired behavior is to return a useful error message.) \begin{enumerate} \item Each process can create at least one communication handle. I.e., if a process executes an {\tt MPI\_INIT\_SEND} or {\tt MPI\_INIT\_RECV} operation, and has no other active handle, then the operation eventually succeeds. It is highly desirable to have generous bounds on the number of concurrently active communication handles each process may have, so that, in practice, {\tt INIT} operations will always be guaranteed to succeed. \item Each process can initiate a communication operation for each active handle. I.e. correct {\tt START} operations always succeed (eventually). \item A send operation is {\bf enabled} if the sending process has issued a {\tt COMPLETE\_SEND} operation and the receiving process has issued a {\tt START} operation for a matching receive. Symmetrically, a receive operation is {\bf enabled} if the receiving process has issued a {\tt COMPLETE\_RECV} operation and the sending process has issued a {\tt START} operation for a matching send. An enabled operation may become {\bf disabled} either because it completes successfully or, in the case of a receive, because the matching message is successfully received by another receive operation. {\bf An enabled operation either completes successfully or becomes permanently disabled.} \item A {\tt FREE} operation always succeeds (eventually). \end{enumerate} The four conditions guarantee progress in the communication subsystem. The third condition guarantee (weak) fairness among competing communication requests. Examples (involving two processors with ids 1 and 2) The following program is safe, and should always succeed. \begin{verbatim} IF (GETID() == 1) { MPI_SEND(buf=sendbuf, len=1000, dest=2, tag=0); MPI_RECV(buf=recvbuf, len=1000, source=2, tag=0); } ELSE \* (GETID() == 2) *\ { MPI_RECV(buf=recvbuf, len=1000, source=1, tag=0); MPI_SEND(buf=sendbuf, len=1000, dest=1, type=0); } \end{verbatim} The following program is erroneous, and should always fail. \begin{verbatim} IF (GETID() == 1) { MPI_SSEND(buf=sendbuf, len=1000, dest=2, tag=0); MPI_SRECV(buf=recvbuf, len=1000, source=2, tag=0); } ELSE \* (GETID() == 2) *\ { MPI_SSEND(buf=sendbuf, len=1000, dest=1, tag=0); MPI_SRECV(buf=recvbuf, len=1000, source=1, tag=0); } \end{verbatim} The send operation of the 1st process can complete only if the matching receive of the second processor is executed; the send operation of the second processor can complete only if the matching receive of the first processor is executed. This program will deadlock. The following program is unsafe, and may succeed or fail, depending on implementation. \begin{verbatim} IF (GETID() == 1) THEN { MPI_SEND(buf=sendbuf, len=1000000, dest=2, tag=0); MPI_RECV(buf=recvbuf, len=1000000, source=2, tag=0); } ELSE \* (GETID() == 2) *\ { MPI_SEND(buf=sendbuf, len=1000000, dest=1, tag=0); MPI_RECV(buf=recvbuf, len=1000000, source=1, tag=0); } \end{verbatim} The message sent by each process has to be copied out before the send operation returns and the receive operation starts. For the program to complete, it is necessary that at least one of the two messages sent is buffered out of either processes' address space. Thus, this program can succeed only if the communication system has sufficient buffer space to buffer 1 MgB of data. If additional requirements will become part of the standard (e.g., bounds on the minimal number of concurrently active handles that need be supported, then further programs become safe. \subsection{Error Handling} \subsubsection{Communication Errors} It is assumed that MPI is implemented on top of an error-free communication subsystem: A message sent is always received correctly, and the user does not need to check for transmission errors, time-outs, and the likes. In other words, MPI does not provide mechanisms to deal with failures in the underlying communication subsystem -- it the responsibility of the MPI implementer to insulate the user from such errors (or to reflect them as global program failures). The same holds true for node failures. The other errors can be divided in two classes: \begin{description} \item[resource errors] Errors that occur in an unsafe programs because of limitations on the resources of the communication subsystem, but would not occur if these subsystem had unbounded resources (buffer overflow, no available handles, etc.) \item[program errors] Errors that result in an erroneous program that will always fail. (send with no matching receive, send to nonexistent destination, etc.) \end{description} Program errors always result in an exception, if detected. MPI will provide two mechanisms for handling resource errors: In {\em system mode} each MPI call may return with a return code that indicates that a resource error was detected; no exception occur, and the program may recover in normal execution mode. In {\em user mode} a detected resource error causes an exception. \discuss{ Program errors cannot always be distinguished from resource errors. Have to decide how to handle doubtful cases in system mode. Need to change the syntax so that all return values (error code excepted) are returned via parameters. It's reasonable to restrict system mode to C and C++ bindings. } \end{document} From owner-mpi-pt2pt@CS.UTK.EDU Mon Feb 8 10:12:00 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA28591; Mon, 8 Feb 93 10:12:00 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA24749; Mon, 8 Feb 93 10:11:20 -0500 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Mon, 8 Feb 1993 10:11:19 EST Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from sampson.ccsf.caltech.edu by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA24741; Mon, 8 Feb 93 10:11:17 -0500 Received: from elephant by sampson.ccsf.caltech.edu with SMTP id AA25516 (5.65c/IDA-1.4.4 for mpi-pt2pt@cs.utk.edu); Mon, 8 Feb 1993 07:11:15 -0800 Received: from lion.parasoft by elephant (4.1/SMI-4.1) id AA06314; Mon, 8 Feb 93 07:06:07 PST Received: by lion.parasoft (4.1/SMI-4.1) id AA16745; Mon, 8 Feb 93 07:10:39 PST Date: Mon, 8 Feb 93 07:10:39 PST From: jwf@lion.Parasoft.COM (Jon Flower) Message-Id: <9302081510.AA16745@lion.parasoft> To: mpi-pt2pt@cs.utk.edu To: mpi-pt2pt@cs.utk.edu Re: Buffering requirements I agree with Jim Cownie's comment that trying to figure out buffering requirements in an application is difficult for the receiver and hence the problems in creating REALLY reliable code with buffered systems. A potentially easier option is the one that Rik Littlefield mentioned - buffering at the sender instead. For most applications it is MUCH easier to compute how much is going to be sent than how much will be received. However, buffering at the sender seems to impose an implementation feature, i.e., that there be an intial message describing the actual transaction followed by a data request and finally the data itself. Is there a way around this? If so, it seems like a great idea to me. If not, is it going to be acceptable to the High Priests to have this three-hop version? Jon Flower From owner-mpi-pt2pt@CS.UTK.EDU Mon Feb 8 10:36:44 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA29236; Mon, 8 Feb 93 10:36:44 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA25820; Mon, 8 Feb 93 10:35:47 -0500 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Mon, 8 Feb 1993 10:35:45 EST Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from sampson.ccsf.caltech.edu by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA25812; Mon, 8 Feb 93 10:35:43 -0500 Received: from elephant by sampson.ccsf.caltech.edu with SMTP id AA25693 (5.65c/IDA-1.4.4 for mpi-pt2pt@cs.utk.edu); Mon, 8 Feb 1993 07:35:40 -0800 Received: from lion.parasoft by elephant (4.1/SMI-4.1) id AA06436; Mon, 8 Feb 93 07:30:33 PST Received: by lion.parasoft (4.1/SMI-4.1) id AA16763; Mon, 8 Feb 93 07:35:05 PST Date: Mon, 8 Feb 93 07:35:05 PST From: jwf@lion.Parasoft.COM (Jon Flower) Message-Id: <9302081535.AA16763@lion.parasoft> To: mpi-pt2pt@cs.utk.edu Subject: /dev/null, /dev/full? To: mpi-pt2pt@cs.utk.edu Re: /dev/null, /dev/full? Despite the fact that we still don't seem to have settled on a model for specifying process identifiers I would like to raise a different issue on this topic ..... the idea of having a special identifier for a node that doesn't actually exist. Sending a message to this destination would always succeed and return as though the data were transmitted, and reading from this node would also return immediately, without modifying the user buffer, but with a return code indicating that no error occurred. I think this offers users great simplifications in their coding. A classic example is the standard domain decomposed PDE solution. In one dimension you have to write code for the guard strip exchange such as if(I_am_on_left_edge) { send_to_right(); read_from_right(); } else if(I_am_on_right_edge) { send_to_left(); read_from_left(); } else if(there_is_more_than_one_node) { send_to_right(); read_from_right(); send_to_left(); read_from_left(); } Note that this code assumes buffered communication. If you have to code unbuffered messages then the logic becomes considerably worse since you have to deal with even and odd parity and the case of an odd number of nodes, etc..... If we have the "no node" concept then the above becomes: send_to_right(); read_from_right(); send_to_left(); read_from_left(); in all cases. Given an "exchange" or "sendrecv" function this code becomes even simpler. When dealing with virtual topologies this is invaluable because the virtual topology code can return this special value as the process I.D. and it can then be passed directly to the communication routines without having to be interpreted at all. I think this can be implemented without cost to the higher level software by making the "no node" identifier illegal for normal communication. Then the basic communication functions which, I imagine, do checking such as if( ! destination_is_legal_node ) { return ILLEGAL_NODE; } can be simply changed to if( ! destination_is_legal_node ) { if( destination_is_no_node ) { /* Pretend this was OK */ return OK; } return ILLEGAL_NODE; } without any impact on the regular code. EXTENSION: Marc Snir raised an interesting point in connection with this idea. Since it is normally used to deal with boundary conditions in the physical world why not extend the idea by allowing a read from the "no node" source to actually implement the user boundary condition either by filling an array with predefined values, or even calling a user-defined function? Again I think this could be implemented at no cost to the higher levels by simply invoking some code in the error handling for normal communication, and extending the "no node" identifier to a range of identifiers. I think in this case you might also be able to run user-supplied code, even when you have a communication co-processor, assuming that the error checking shown above occurs early enough in the sequence that it would still be in the main processor(?) I don't have any experience with this extension but it certainly seems "cute". One possible objection might be that the boundary case code is almost certainly already coded in any existing application and pushing it down into the message passing layer might be a step backwards. For applications written from scratch, however, it is a nice abstraction. Jon Flower From owner-mpi-pt2pt@CS.UTK.EDU Mon Feb 8 10:36:54 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA29242; Mon, 8 Feb 93 10:36:54 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA25849; Mon, 8 Feb 93 10:36:36 -0500 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Mon, 8 Feb 1993 10:36:34 EST Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from cs.sandia.gov by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA25840; Mon, 8 Feb 93 10:36:33 -0500 Received: from panther.cs.sandia.gov by cs.sandia.gov (4.1/SMI-4.1) id AA20721; Mon, 8 Feb 93 08:36:26 MST Received: by panther.cs.sandia.gov (Smail3.1.28.1 #1) id m0nLaX2-0016ZKC; Mon, 8 Feb 93 08:36 MST Message-Id: Date: Mon, 8 Feb 93 08:36 MST From: srwheat@cs.sandia.gov (Stephen R. Wheat) To: mpi-pt2pt@cs.utk.edu Subject: Re: Buffering requirements Jon asks: >> Is there a way around this? If so, it seems like a great idea >> to me. If not, is it going to be acceptable to the High Priests >> to have this three-hop version? This brings us back to the question of the "Ready Receiver". That is, should one KNOW that a matched receive has already been posted, then one should be able perform the send without having to go through the heavy duty "RTS/CTS/Data" sequence. One other point I would like to raise is that the phrase "system buffering" implies an implementation where the system provides the buffering rather than the application. I believe that one could argue that "application buffering" would be more portable, in that the application/library would depend on more basic services, such as application memory allocation to provide the buffering. Stephen From owner-mpi-pt2pt@CS.UTK.EDU Mon Feb 8 10:54:41 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA29472; Mon, 8 Feb 93 10:54:41 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA26520; Mon, 8 Feb 93 10:53:53 -0500 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Mon, 8 Feb 1993 10:53:51 EST Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from sampson.ccsf.caltech.edu by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA26512; Mon, 8 Feb 93 10:53:49 -0500 Received: from elephant by sampson.ccsf.caltech.edu with SMTP id AA25873 (5.65c/IDA-1.4.4 for mpi-pt2pt@cs.utk.edu); Mon, 8 Feb 1993 07:53:47 -0800 Received: from lion.parasoft by elephant (4.1/SMI-4.1) id AA06638; Mon, 8 Feb 93 07:48:40 PST Received: by lion.parasoft (4.1/SMI-4.1) id AA16875; Mon, 8 Feb 93 07:53:12 PST Date: Mon, 8 Feb 93 07:53:12 PST From: jwf@lion.Parasoft.COM (Jon Flower) Message-Id: <9302081553.AA16875@lion.parasoft> To: mpi-pt2pt@cs.utk.edu Subject: System vs. Application buffers Having application level buffers certainly seems to give the user more control over the amount of memory they hve to play with but is it consistent with layered (library) packages? Assume that we have a simple three-tier library calling sequence with buffer requirements as follows User-application (Needs buffer space X) Library-1 (Needs buffer space Y) Library-2 (Needs buffer space Z) Are we then going to ask for memory (X+Y+Z) or simply max(X,Y,Z)? I suspect that if the system is buffering we'll probably get the latter. In the application buffered case you could probably have either if buffers are allocated with malloc/realloc. But if buffers are declared explicitly then X+Y+Z is probably going to be required. One option would be to have one of the arguments to the library package be its message buffer space in the same way that IMSL routines have you pass in workspace as arguments. Then you could potentially re-use an outer layer buffer. This seems ugly but would probably work. Jon From owner-mpi-pt2pt@CS.UTK.EDU Mon Feb 8 12:19:09 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA01665; Mon, 8 Feb 93 12:19:09 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA00736; Mon, 8 Feb 93 12:18:15 -0500 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Mon, 8 Feb 1993 12:18:14 EST Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from pnlg.pnl.gov by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA00728; Mon, 8 Feb 93 12:18:11 -0500 Received: from carbon.pnl.gov (130.20.65.121) by pnlg.pnl.gov; Mon, 8 Feb 93 09:04 PST Received: from sodium.pnl.gov by carbon.pnl.gov (4.1/SMI-4.1) id AA23874; Mon, 8 Feb 93 09:03:34 PST Received: by sodium.pnl.gov (4.1/SMI-4.0) id AA18935; Mon, 8 Feb 93 09:03:26 PST Date: Mon, 8 Feb 93 09:03:26 PST From: d39135@sodium.pnl.gov Subject: Re: Buffering requirements To: jwf@lion.Parasoft.COM, mpi-pt2pt@cs.utk.edu Cc: d39135@sodium.pnl.gov Message-Id: <9302081703.AA18935@sodium.pnl.gov> X-Envelope-To: mpi-pt2pt@cs.utk.edu Jon Flower says: > I agree with Jim Cownie's comment that trying to figure out buffering > requirements in an application is difficult for the receiver and hence > the problems in creating REALLY reliable code with buffered systems. > A potentially easier option is the one that Rik Littlefield mentioned - > buffering at the sender instead. For most applications it is MUCH > easier to compute how much is going to be sent than how much will > be received. > > However, buffering at the sender seems to impose an implementation > feature, i.e., that there be an intial message describing the actual > transaction followed by a data request and finally the data itself. > > Is there a way around this? If so, it seems like a great idea > to me. If not, is it going to be acceptable to the High Priests > to have this three-hop version? Let's see if I can remember what it was I proposed. I think it went like this: 1. Provide an MPI hook that an application could call to declare the maximum amount of data that might be sent to unready receivers. 2. Require that an application calling the hook also provide buffer space for that much data. 3. Require that if the hook is called, MPI then must *act as if* it were working in "sender-buffers" mode. 4. One possible implementation of this approach is for MPI to simply transform all blocking send calls into . copy data to (application-provided) buffer space . issue non-blocking send from the buffer copy . return to application and . check completion on subsequent MPI call(s) This implementation does not seem to impose any extra costs at the protocol level. In particular, if non-blocking send can be accomplished without an "RTS/CTS/data" sequence, say by assuming some amount of buffering at the receiver as in the Intel iPSC short-message and DELTA protocols, then the sender-buffers implementation proposed here should be able to do the same. No doubt many optimizations within MPI are possible -- the proposal just requires that MPI act as described here. I am not sure how the sender-buffers declaration should look in order to best support libraries. One possibility would be to allow the declarations to be nested, so that each library could declare its own needs at the point where those needs are determined. I'm not convinced that this is much easier than just (re)coding the library to use non-blocking sends directly, but it seems to fit with the basic idea of portable support for non-blocking csend's. --Rik Littlefield From owner-mpi-pt2pt@CS.UTK.EDU Tue Feb 9 02:04:34 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA18444; Tue, 9 Feb 93 02:04:34 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA06135; Tue, 9 Feb 93 02:03:07 -0500 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Tue, 9 Feb 1993 02:03:05 EST Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from antares.mcs.anl.gov by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA06127; Tue, 9 Feb 93 02:03:03 -0500 Received: from donner.mcs.anl.gov by antares.mcs.anl.gov (4.1/SMI-GAR) id AA20719; Tue, 9 Feb 93 01:02:59 CST Received: by donner.mcs.anl.gov (4.1/GCF-5.8) id AA04746; Tue, 9 Feb 93 01:02:55 CST Message-Id: <9302090702.AA04746@donner.mcs.anl.gov> To: mpi-pt2pt@cs.utk.edu Subject: A suggestion for a multi-level MPI Date: Tue, 09 Feb 93 01:02:54 CST From: Rusty Lusk Last week Tom Henderson asked for more detail about the multiple-level proposal, so we wrote up a slightly more detailed explanation of the idea, with some examples, which is appended at the end of this note. It is partly subsumed by Marc Snir's posting of a thoroughly worked out point-to-point proposal, which has a lot in common with what we described as Levels 1 and 2. A few notes: In Level 1, we use a very minimal set of required parameters in the "init" calls, so that more of the parameters can be reset inside a loop without acquiring a new handle. We really think that Levels 3 and 4 (the simpler levels, where there are fewer options and parameters) are necessary in order to provide a simple interface for basic users. We didn't change the way heterogeneity is handled, but Tom Henderson's suggestion (that the datatypes *always* be specified, perhaps with a BYTES type for the "raw" datatype) is a quite sensible one. Here is a medium-sized description of what we mean by a "multi-level" standard. The specific routines and their parameters are not as important as the idea that complexity can be managed in this way. ---------------------------------------------------------------------- \documentstyle[11pt]{article} \textwidth=6.0in \textheight=9.0in \hoffset=-0.5in \voffset=-1.0in \parskip=.1in \def\explicitspace{\hbox{\vrule height 2pt width .5pt \vbox to 8pt{\vfil\hrule height .5pt width 6pt}% \vrule height 2pt width .5pt}} \begin{document} \title{\Large \bf A Proposal for a Multi-Level Message-Passing Interface Standard} \author{ {\it William D. Gropp and Ewing L. Lusk} \\ Argonne National Laboratory \\ Argonne, IL 60439 \\ gropp@mcs.anl.gov \\ lusk@mcs.anl.gov } \date{} \maketitle \begin{center} {\large \it DRAFT} \end{center} \section{Introduction} \label{sec:introduction} In \cite{gropp-lusk:mpi1} we described the organization of a library of routines that could be used to implement a standard set of message-passing functions. We used the library to rapidly implement most of the routines described in \cite{dongarra-hempel-hey-walker:october-draft}. The implementation testbed was not originally offered as a counterproposal to the routines described in \cite{dongarra-hempel-hey-walker:october-draft} since no real discussion had taken place in the MPI Committee yet, but it did contain many capabilities we believed would ultimately be desired. It was constructed in a way that offered a wide range of capabilities with a minimum of complexity in syntax. Basically, it consisted of a large number of separate routines whose names were orthogonally organized along a set of axes corresponding to synchronization type, buffer structure, high-performance mode, and heterogeneity specification. See \cite{gropp-lusk:mpi1} for details. At the MPI meeting in January, it became apparent that \begin{itemize} \item Many of the committee members were uncomfortable with having a large number of routines. \item A very wide range of capabilities and options was nonetheless desired. \item A very wide range of users was to be targeted, from experienced users requiring access to high-performance operations to beginning users requiring simplicity in both syntax and semantics. \item While discussion of precise syntax was deferred, it might be useful to suggest one, in order that everyone could see what the ultimate standard might look like. \end{itemize} We added a further consideration of our own: \begin{itemize} \item The standard needs a way to expand in order to add functionality or options that have not been envisioned yet, without becoming incompatible with itself. \item An implementation would have to be extremely adaptable if it was expected to be able to track the discussions of the MPI Committee. \end{itemize} The present draft is a reflection of these considerations. It is still incomplete (as are the committee's discussions) but is offered in the hope that it might be useful. It focuses primarily on the point-to-point operations. \section{The Basic Idea: Orthogonality of Function} \label{sec:orthogonality} The MPI committee has discussed a large number of options for point-to-point message passing. The fact that there could be as many as a thousand different ``send'' routines has become a joke, and some believe that the scope of MPI way have to restricted in order that it not become unwieldy. This problem has at least two solutions, and we explore each of them here. What makes the porblem tractable is that the options that are being discussed are {\it orthogonal\/}; that is, choice of one value of an option does not determine what value another opetion must have. For example, whether a send is blocking or nonblocking is completely independent of whether the message consists of contiguous memory or not. This orthogonality can be exploited to make the interface simple to specify, implement, and use, while at the same time not sacrificing flexibility. The first way is to set the options with a ``universal'' {\tt set\_option(handle,option,value)} routine. There will be a small number of allowed values for {\tt option}, and for most of these, a small number of values for {\tt value}. (This is the approach taken in Level I, below.) The second way is to absorb the options and their values into the routine names. This leads to a large {\em number\/} of routines, but their names can be made predictable and descriptive, so that they are easy to understand and use. (This is the approach taken in Level II, below.) The principal organizing idea of this library is that it proposes multiple sets of routines, roughly grouped by complexity level, {\it all of which are proposed for simultaneous inclusion as part of the MPI Standard}. That is, although it is possible to implement some of the higher-level routines in terms of the lower ones, either by combining short sequences of low-level calls or by supplying specific values for parameters, it is strongly suggested here that all levels be adopted together. This is the way to attain a set of routines that, taken altogether, offers complete flexibility for the advanced user while not imposing any unnecessary complexity in either syntax or semantics on the beginner. The relationships among the levels remain extremely useful for defining the semantics of the higher-level routines in terms of the lower level ones, which have simpler semantics. We now describe the four ``levels'' of MPI routines. For expository purposes, we describe them in what might be considered an unnatural order: \begin{description} \item[Level IV] The highest level, consisting of only four routines, suitable for beginning users. \item[Level I] The lowest level, giving complete access to all of the facilities offered by the MPI Standard. The actual number of routines is small. \item[Level III] A middle level, required in order to write programs that are more efficient and portable than those expressible with Level IV. This is the approximate level of most current message-passing systems. \item[Level II] A level between I and III, chosen for convenience of use without giving up the features of Level I and possible added efficiency in the implementation. It corresponds roughly to the set of routines described in \cite{gropp-lusk:mpi1}. \end{description} It is to be emphasized that this layered structure is for ease of definition and convenience of use, and does not imply a layered structure for the implementation, which could result in suboptimal efficiency. Likewise, it is not suggested here that a user would have to pick one of the layers and stick with it. In all cases where it makes sense, the layers should be interoperable; that is, routines from all layers may appear in the same program. \section{The Point-to-point MPI routines} \label{sec:pt-to-pt} Note on syntax: we have to pick {\em some\/} syntax in order to write these down clearly. We leave the selection of an elegant, expressive, complete, intuitive syntax and appropriate language bindings to the further discussions of the MPI Committee. \subsection{Level IV: The Four-function Interface} \label{sec:four-func} A large number of parallel algorithms can be expressed in terms of four simple functions. In the quest for simplicity, it is useful to see how far the standard might be able to go. It appears to us that the following set of routines represents a minimum. (Strictly speaking, only two of these belong in this point-to-point section, but it is not worth separating this tiny set of routines into parts.) \begin{verbatim} MPI_numpids() returns the number of active process ID's MPI_mypid() returns the id of the calling process MPI_send(dest,tag,buffer,length) sends the contiguous set of length bytes addressed by buffer to process dest and tags the message with tag. The buffer is available for reuse when this returns. MPI_recv(dest,tag,buffer,length) waits for a message from process dest with tag tag and puts in buffer, truncating it to length if necessary for it to fit into buffer. \end{verbatim} More precise definitions of these can be made once we have defined the lower levels. Basically, the point-to-point component consists of an asynchronous send that blocks only until the buffer can be reused, and a blocking receive that waits for the arrival of a message. \subsection{Level I: Access to complete functionality in a small number of routines} \label{sec:level1} The above trivial interface is enough to specify a parallel program, and in many cases, is actually sufficient. The only reason for adding more point-to-point routines is to improve efficiency, portability, and convenience. Before defining what perhaps will be the most commonly-used layer for most programmers (Level III), we go down to Level I, where all of the MPI capabilities will be exposed. We can then describe the intermediate layers more easily, in terms of the lowest one. The idea here is to fully exploit for the lowest level two ideas that were discussed favorably at the January meeting: \begin{itemize} \item separation of the ``setup'' part of an operation from the ``initiate'' part. This idea is described in detail in \cite{snir:notes}. \item the use of a control structure (either in system space or user space) to hold the parameters for an operation that has been ``set up'', but is not complete. (It may or may not have been initiated.) This structure serves as the ``handle'' for the operation. Its precise format depends on the MPI implementation, and it is accessed by the user only through MPI routines. \end{itemize} \begin{verbatim} handle = MPI_init_send(dest,tag) handle = MPI_init_recv() MPI_mod_send(handle,option,value) MPI_mod_recv(handle,option,value) MPI_do_send(handle) MPI_do_recv(handle) MPI_free(handle) \end{verbatim} The idea is that the ``init'' routines return a handle that is usable {\em as is\/}, because it selects a reasonable set of defaults, but that can be modified before the operations is actually initiated. If the {\tt do\_send} or {\tt do\_recv} operations are issued without any intervening calls to {\tt mod} routines, then the default behavior occurs, which we will have to decide on. However a wide range of options can be set by the {\tt MPI\_mod\_send} and {\tt MPI\_mod\_recv} routines. Although we are not intending to specify an implementation, it will perhaps clarify things to envision how this might work. The {\tt init} routines will allocate a block of storage (or, alternatively, deal with a block a storage allocated by the user). The ``do'' routines will take their parameters from this control block. The control block will be immediately usable, since the minimal parameters will be specified on the ``init'' call. (A send doesn't make sense without a destination and a tag, so they are its parameters. A receive need specify no filtering by tag or sender, and could receive a message of 0 bytes, requiring no buffer, hence the {\tt recv\_init} has no parameters.) The {\tt MPI\_mod\_send} and {\tt MPI\_mod\_recv} are used to modify the default parameters set up by the {\tt MPI\_init\_send} and {\tt MPI\_init\_recv} routines. The parameters to the {\tt MPI\_mod} routines are as follows: \begin{description} \item[handle] the value returned by a previous {\tt MPI\_init\_send} or {\tt MPI\_init\_recv}. \item[option] one of a collection of symbolic constants specifying the parameter being set. \item[value] one of a collection of symbolic constants or integers or addresses, depending on the option. \end{description} One suggestion for a set of options and their allowed values might be: \begin{verbatim} option value ------- ------- BUFTYPE - one of CONTIG, STRIDED, VEC, HVEC BLOCKING - one of NONE, LOCAL, GLOBAL TRANSLAT - one of NONE, XDR PROTOCOL - one of PLAIN, RCVRRDY BUFADD - address of buffer, or NULL BUFLEN - integer lengh of buffer DATATYPE - one of INT, FLOAT, DOUBLE, LONGINT, BYTES, CHAR TAG - integer or (on receive) ANY BUFVEC - address of structure describing the buffer data, or NULL SOURCE - (on receive) integer process id, or ANY \end{verbatim} The {\tt MPI\_init\_send(dest,tag)} could reasonably set the following defaults, in addition to setting the {\tt dest} and {\tt tag} fields in the control block: \begin{verbatim} BUFTYPE = CONTIG BLOCKING = LOCAL TRANSLAT = NONE PROTOCOL = PLAIN BUFADD = NULL BUFLEN = 0 DATATYPE = BYTES BUFVEC = NULL \end{verbatim} The {\tt MPI\_init\_recv()} could set the same default values, together with {\tt SOURCE = ANY} and {\tt TAG = ANY}. (In a real proposed standard, these symbolic constants should have an MPI prefix, in order to prevent the name-space pollution problem, but here we leave them off for simplicity.) The large number of routines listed in \cite{gropp-lusk:mpi1}, some of which will reappear in Level II, have in Level IV been absorbed into parameter values of the {\tt MPI\_mod\_send} and {\tt MPI\_mod\_recv} routines, in particular the first four values of {\tt option}. The advantages of this approach are: \begin{itemize} \item a small number of routines at the lowest level \item extensibility: new options, even implementation-specific ones, can be handled by adding more options without adding more routines. \end{itemize} The disadvantage is that to send a message requires a sequence of subroutine calls, adding both program complexity and a moderate amount of overhead to each operation. It is this problem that is addressed by level II. \subsection{Level III: A Medium-Level Interface} \label{sec:level3} In this section we describe a level that is designed to provide access to a sufficient set of MPI routines that efficient, portable programs may be written in it, without accessing all of the flexibility of the complete MPI interface. This level will, however, support both blocking and non-blocking sends and receives as well as messages between machines from different vendors. By imposing a few restrictions on what can be done at this level, however, we can reduce the number of routines and shorten parameter lists, thus reducing complexity. A possible set of restrictions for this level might be: \begin{itemize} \item No noncontiguous buffer structures \item No mixed datatypes in messages destined for heterogeneous processors \item No ``rendezvous'' send \item No ``receiver-ready'' messages \end{itemize} Programmers who need more functionality than is offered at this level can use the routines in levels I or II. The set of Level III routines, given the above restrictions, looks like \[ \left[ \begin{array}{c} n \\ b \end{array} \right] \left[ \begin{array}{c} send \\ recv \end{array} \right] \left[ \begin{array}{c} \explicitspace \\ h \end{array} \right], \] \noindent together with the {\tt wait} and {\tt status} routines. The first column specifies whether the opertion is to block or not, and the last column specifies whether the message is to be translated for transmission to a machine with a different data representation. That is, the above diagram ellustrates the way the names are to made up. Since in this case the list is relatively short, we can also list them: \begin{verbatim} bsend(tag,dest,bufadd,buflen) nsend(tag,dest,bufadd,buflen) bsendh(tag,dest,bufadd,datatype,numitems) nsendh(tag,dest,bufadd,datatype,numitems) brecv(tag,dest,bufadd,buflen) nrecv(tag,dest,bufadd,buflen) brecvh(tag,dest,bufadd,datatype,numitems) nrecvh(tag,dest,bufadd,datatype,numitems) \end{verbatim} \subsection{Level II: Flexibility and Efficiency in a large number of routines} \label{sec:level2} For Level II, we use only the naming schematic, since the list becomes rather long. The first column represents the choice of buffer structure ({\tt BUFTYPE} in Level I). The second column specifies the blocking option ({\tt BLOCKING}). Then whether it is a send or receive, then the {\tt TRANSLAT} option, and finally the receiver-ready protocol ({\tt PROTOCOL = RCVRRDY}). \[ \left[ \begin{array}{c} c \\ s \\ g \end{array} \right] \left[ \begin{array}{c} n \\ b \\ s \end{array} \right] \left[ \begin{array}{c} send \\ recv \end{array} \right] \left[ \begin{array}{c} \explicitspace \\ h \end{array} \right] \left[ \begin{array}{c} \explicitspace \\ rr \end{array} \right], \] \noindent See \cite{gropp-lusk:mpi1} for details. \section{Examples of the Multi-Level Approach} \label{sec:examples} Before giving the details, we give three examples of how certain operations would be expressed at teach of the levels. Note that some operations are not available at the higher levels. The point is that simple operations should be simple to express while more sophisticated ones may be more complicated. \paragraph{Example 1.} A blocking send of a contiguous stream of uninterpreted bytes. At Level IV (the four-function level) nothing but the basic parameters needs to specified, since this is the only type of send operation. \begin{verbatim} MPI_send(dest,tag,buf,len) \end{verbatim} At Level III, where both blocking and non-blocking sends are possible, we need to specify that this is a blocking send. We choose to incorporate the specification into the routine name. \begin{verbatim} MPI_sendb(dest,tag,buf,len) \end{verbatim} At Level II, noncontiguous buffers are allowed, so we need to specify that the buffer is contiguous, which we again do with the name. \begin{verbatim} MPI_csendb(dest,tag,buf,len) \end{verbatim} At Level I, we need to set up the operation first, then modify the default parameters of the operation before actually initiating it. \begin{verbatim} handle = MPI_init_send(dest,tag) MPI_mod_send(handle,BUFADD,buf) MPI_mod_send(handle,BUFLEN,len) MPI_mod_send(handle,BLOCKING,LOCAL) MPI_do_send(handle) MPI_send_wait(handle) MPI_send_free(handle) \end{verbatim} \paragraph{Example 2.} A non-blocking receive of data that is all of the same type into a contiguous buffer, with translation for receipt from a process on a machine with a possibly different data representation. At Level IV, this operation is impossible to express. At Level III, the specification that is it is non-blocking and heterogeneous is encoded in the name, which specifies a routine that expects a datatype and number of items, rather than a length. \begin{verbatim} handle = MPI_nrecvh(source,tag,buf,datatype,numvals) \end{verbatim} At Level II, we must also specify that the buffer is contiguous. \begin{verbatim} handle = MPI_cnrecvh(source,tag,buf,datatype,numvals) \end{verbatim} At Level I, we set up the operation and then modify its parameters. \begin{verbatim} handle = MPI_init_recv() MPI_mod_recv(handle,SOURCE,source) MPI_mod_recv(handle,TAG,tag) MPI_mod_recv(handle,BUFADD,buf) MPI_mod_recv(handle,BUFTYPE,CONTIG) MPI_mod_recv(handle,TRANSLATE,XDR) MPI_mod_recv(handle,DATATYPE,datatype) MPI_mod_recv(handle,NUMVALS,numvals) MPI_do_recv(handle) MPI_recv_status(handle) \end{verbatim} \paragraph{Example 3.} A non-blocking send of non-contiguous, mixed heterogeneous data, with the assumption that the receiver has already issued his receive. At Levels IV and III, this is impossible to express. At Level II, most of the options are encoded in the name. \begin{verbatim} MPI_gsendhrr(dest,tag,...) \end{verbatim} At Level I, we use the same basic routines to set the options. \begin{verbatim} handle = MPI_init_send(dest,tag) MPI_mod(handle,BUFTYPE,HVEC) (Build vector describing data locations, types, and numbers of items in each clump.) MPI_mod_send(handle,HVECADD,vec) MPI_mod_send(handle,TRANSLATE,XDR) MPI_mod_send(handle,RECVRRDY,TRUE) MPI_do_send(handle) \end{verbatim} \paragraph{Example 4.} Set up a channel from one process to another, and then use it for repeated send operations of different data. This operation cannot be specified at Levels II, III, or IV. At Level I, it can be specified in the following way. \begin{verbatim} handle = MPI_init_send(dest,tag) begin loop MPI_mod_send(handle,BUFADD,buf) MPI_mod_send(handle,BUFLEN,len) MPI_do_send(handle) end loop \end{verbatim} \section{Collective Communication} \label{sec:collective} The idea of a separate setup routine, together with multiple levels to accomodate both and basic operations as well as more elaborate ones, should prove useful for collective operations as well. We will try to extend the point-to-point operations to collective operations in as natural a way as possible. \bibliographystyle{plain} \begin{thebibliography}{1} \bibitem{dongarra-hempel-hey-walker:october-draft} J.~Dongarra, R.~Hempel, T.~Hey, and D.~Walker. \newblock A proposal for a message passing interface standard. \newblock (circulated before November meeting). \bibitem{gropp-lusk:mpi1} Bill Gropp and Rusty Lusk. \newblock A test implementation of the {MPI} draft message-passing interface standard. \newblock Technical Report ANL--92--47, Argonne National Laboratory, December 1992. \newblock (circulated at January meeting). \bibitem{snir:notes} Marc Snir. \newblock Message-passing interface--outline. \newblock (circulated at January meeting). \end{thebibliography} \end{document} From owner-mpi-pt2pt@CS.UTK.EDU Mon Feb 15 06:33:47 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA22237; Mon, 15 Feb 93 06:33:47 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA11881; Mon, 15 Feb 93 06:32:40 -0500 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Mon, 15 Feb 1993 06:32:39 EST Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from daedalus.epcc.ed.ac.uk by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA11873; Mon, 15 Feb 93 06:32:34 -0500 Date: Mon, 15 Feb 93 11:32:26 GMT Message-Id: <21543.9302151132@subnode.epcc.ed.ac.uk> From: L J Clarke Subject: various To: mpi-pt2pt@cs.utk.edu Reply-To: lyndon@epcc.ed.ac.uk Dear colleagues I circulated the documents prepared by Marc, and Bill & Rusty, around interested people locally, for discussion. Here are some resulting considerations of the point-to-point component of MPI. a) Synchronous messages and blocking/nonblocking communication calls: The properties of end to end synchronisation, and blocking/nonblocking communication calls are "orthogonal" in the sense described by Bill and Rusty. Thus, another "axis" in the procedure naming (section 3.3) should be added, which perhaps is either nothing or 's'. The approach taken by Marc is a little, different and appears to implicitly recognise this. This doesnt seem (to me) to fit in with the framework which Marc uses, but it might be appropriate to consider synchronous and ready-receiver as points on the same axis. They both refer to the knowledge which the sender has about receipt of the message. In the synchronous case, the sender absolutely knows that the receiver has got the message; in the ready-receiver case the sender (presumably) has no knowledge of whether the receiver has or has not got the message. This approach would also remove the curious "synchronous ready receiver" from Marc's document. b) Cancellation of nonblocking communications: This subject has appeared before. We strongly support the points made by Jim Cownie regarding the requirement for cancellation of nonblocking communications. An example of its usefullness arises when writing libraries encapsulating common communication patterns, we view it as desirable that the communication call should be able to return with an error code if something is wrong. A routine XXX may not be able to detect this until after some nonblocking communications have been started, and some of those which have been started have also been completed and the information therein examined. At this point it becomes known that the operation cannot succeed, however there are these nonblockinf communications outstanding. It may not be possible to wait for them to complete, as they may never complete since the matching operation may never be started, so th natural thing to do is cancel the outstanding communications and return cleanly with an error code. The importance of the clean return is apparent when one considers the possibility that the memory into which pending nonblocking receives may write (or may not, unknown) is a local variable of the caller of XXX, which will itself then wish to return to its caller, which will in turn call some user error procedure. In this case the stack can become corrupted by data from the uncancelled nonblocking receive started in XXX and the whole program barfs up. Not nice. c) Matching of buffer structures: The proposals contain three provisions for buffer structures: contiguous, strided and iovec (ignoring heterogeneous issues for the moment, see below). Each of these should be defined to provide a byte stream (which may not map to a contiguous set of bytes in memory), and the programmer should thus be allowed to mix'n'match sends and receives of different types freely. This provides a handy way of performing scatter and gather type operations. d) Heterogeneous issues and data conversion: Provision of data type conversion in MPI is gaining support. This makes sense to me, since a lot of PVM users are (presumably) taking advantage of such provisions in PVM. The proposal of Bill and Rusty contains a set of routines which provide data type conversion. Unfortunately the capabilities provided are weaker than PVM, since they allow data conversion in messages of the form "array of datatype" only. This raises the issue of user defined data types in C/Fortran90 (of course there is no issue in Fortran77). At least with the PVM interface the programmer can portably communicate such data types, albeit with a little effort and overhead. e) Completion/checking functions: In sections 1.5.5 and 1.5.6 of Marc's document, the START and COMPLETE operations are described. I cannot agree that there should be seperate MPI_SEND_XXX and MPI_RECV_XXX routines (XXX is one of {START,COMPLETE,CHECK}). The handle contains all of the information regarding the kind of operation which has been started, so it is not necessary to have different routines. It can be most convenient to be able to wait for a mixture of sends and receives to complete, more specifically to wait for any of such a mixture. In order to do this, you must have an interface which allows send and receive handles to be passed to the same wait routine. Another advatage, perhaps, is that htis wait function does not have to raise an error if the wrong kind of handle is supplied, although this advantage is minimal since it will have to check validity of handles in a more general sense. There has been an argument of readability for having seperate routines for send and receive. We could resolve this by defining MPI_SEND_XXX and MPI_RECV_XXX to be wrappers around MPI_XXX, which check that all handles are of the appropriate nature. Best Wishes Lyndon /--------------------------------------------------------\ e||) | Lyndon J Clarke Edinburgh Parallel Computing Centre | e||) c||c | Tel: 031 650 5021 Email: lyndon@epcc.edinburgh.ac.uk | c||c \--------------------------------------------------------/ From owner-mpi-pt2pt@CS.UTK.EDU Mon Feb 15 10:09:46 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA04339; Mon, 15 Feb 93 10:09:46 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA20259; Mon, 15 Feb 93 10:08:42 -0500 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Mon, 15 Feb 1993 10:08:41 EST Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from antares.mcs.anl.gov by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA20251; Mon, 15 Feb 93 10:08:40 -0500 Received: from godzilla.mcs.anl.gov by antares.mcs.anl.gov (4.1/SMI-GAR) id AA15846; Mon, 15 Feb 93 09:08:27 CST From: gropp@antares.mcs.anl.gov (William Gropp) Received: by godzilla.mcs.anl.gov (4.1/GeneV4) id AA27168; Mon, 15 Feb 93 09:08:24 CST Date: Mon, 15 Feb 93 09:08:24 CST Message-Id: <9302151508.AA27168@godzilla.mcs.anl.gov> To: lyndon@epcc.ed.ac.uk Cc: mpi-pt2pt@cs.utk.edu In-Reply-To: L J Clarke's message of Mon, 15 Feb 93 11:32:26 GMT <21543.9302151132@subnode.epcc.ed.ac.uk> Subject: various X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Mon, 15 Feb 1993 06:32:39 EST Date: Mon, 15 Feb 93 11:32:26 GMT From: L J Clarke Reply-To: lyndon@epcc.ed.ac.uk ... Just one clarification: ... d) Heterogeneous issues and data conversion: Provision of data type conversion in MPI is gaining support. This makes sense to me, since a lot of PVM users are (presumably) taking advantage of such provisions in PVM. The proposal of Bill and Rusty contains a set of routines which provide data type conversion. Unfortunately the capabilities provided are weaker than PVM, since they allow data conversion in messages of the form "array of datatype" only. Actually, our proposal is somewhat stronger than this, at least at the "expert" level (level 1): The user can provide an array of structures, each of which is essentially an "iovec" (that is, pointer and length) with an additional field for the datatype. Thus, a message consisting of a pair of integers and an array of doubles could be sent as (please excuse the syntax; this can be cleaned up in any number of ways): mvec[0].len = 2 * sizeof(int); mvec[0].ptr = &i[0]; mvec[0].datatype = MPI_INT; mvec[1].len = 347 * sizeof(double); mvec[1].ptr = &array[0]; mvec[1].datatype = MPI_DOUBLE; Alternate forms of this structure can basically define a "channel program" that allows for things like non-unit-stride access and scatter/gathers. Thus, just as with PVM, a single message may portably contain multiple data types; the difference is in when the data is presented to the interface (one datatype at a time, as in PVM, or all at once, as in our proposal). This raises the issue of user defined data types in C/Fortran90 (of course there is no issue in Fortran77). At least with the PVM interface the programmer can portably communicate such data types, albeit with a little effort and overhead. This is our intent as well. Note that it would be easy to write a tool that would take a structure/derived type definition and generate the appropriate setup/calls in either the PVM case or our proposal (A compiler could do it to, but I suspect that initially many would be happy not to have the compiler tinkered with, particularly since such enhancements would not be portable). Bill and Rusty From owner-mpi-pt2pt@CS.UTK.EDU Mon Feb 15 10:21:52 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA04647; Mon, 15 Feb 93 10:21:52 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA20871; Mon, 15 Feb 93 10:21:27 -0500 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Mon, 15 Feb 1993 10:21:25 EST Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from daedalus.epcc.ed.ac.uk by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA20820; Mon, 15 Feb 93 10:20:39 -0500 Date: Mon, 15 Feb 93 15:19:51 GMT Message-Id: <21840.9302151519@subnode.epcc.ed.ac.uk> From: L J Clarke Subject: Re: various To: gropp@antares.mcs.anl.gov (William Gropp) In-Reply-To: William Gropp's message of Mon, 15 Feb 93 09:08:24 CST Reply-To: lyndon@epcc.ed.ac.uk Cc: mpi-pt2pt@cs.utk.edu Hi Bill and Rusty Thanks for the clarification. I hadn't picked up the 'datatype' field in the descriptor structure for iovec messages - I can't find it in the proposal document. What you propose here for level 1 is indeed as strong as PVM - it may be more convenient also. Of course the iovec is then a bit less like the kind of iovec that Un*x programmers are familiar with, and used in this way it makes sense to check that the iovec structures at sender and receiver have some kind of consonance. Best Wishes Lyndon /--------------------------------------------------------\ e||) | Lyndon J Clarke Edinburgh Parallel Computing Centre | e||) c||c | Tel: 031 650 5021 Email: lyndon@epcc.edinburgh.ac.uk | c||c \--------------------------------------------------------/ From owner-mpi-pt2pt@CS.UTK.EDU Wed Mar 3 15:26:59 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA08992; Wed, 3 Mar 93 15:26:59 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA03642; Wed, 3 Mar 93 15:25:54 -0500 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Wed, 3 Mar 1993 15:25:53 EST Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from [129.215.56.21] by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA03627; Wed, 3 Mar 93 15:25:50 -0500 Date: Wed, 3 Mar 93 20:25:31 GMT Message-Id: <6861.9303032025@subnode.epcc.ed.ac.uk> From: L J Clarke Subject: safe, unsafe and erroneous MPI programs To: mpi-pt2pt@cs.utk.edu Reply-To: lyndon@epcc.ed.ac.uk MPI, more specifically the point-to-point component, describes three classes of programs: safe, unsafe and erroneous. I'm thinking about buffer resource exhaustion here. It is possible to write safe programs in MPI. It is not possible to identify a subset of MPI which is guaranteed to provide either safe or erroneous programs, and cannot produce unsafe programs. We think that MPI should contain such an identifiable subset. This subset was provided by the SYNCHRONOUS completion mode proposed by Marc Snir, which was rejected at the last meeting. I could not support the proposal because I do not believe that the completion mode proposed is actually what users want, and did not reflect common practice. Primary proposal ---------------- I propose that a further completion mode should be added to MPI, in addition to the surviving REGULAR completion mode, which reflects common practice and creates the identifiable subset of MPI which. I propose that a SECURE (for want of a better name) completion mode be added, which has the following semantics: (a) The COMPLETE operation will wait until the user buffer for send is clear for use. In send the data has been copied from the user buffer (without specification of where it has been copied to), and in receive the data has been copied to the user buffer. AND (b) The COMPLETE operation will wait until the sender (receiver) has performed the START operation which matches the receiver (sender). Note that (a) is just the REGULAR completion mode, and (b) has no additional meaning for a receiver. It really just applies to senders. With this addition, the identifiable subset of MPI which produces either safe or erroneous programs is that which contains the SECURE complete mode and does not contain the READY start mode. The program Process 1 Process 2 --------- --------- secure send to 2 secure send to 1 blah blah blah secure recv from 2 secure recv from 1 is erroneous (by definition?). It will deadlock with every correct implementation of MPI. The program Process 1 Process 2 --------- --------- nonblocking secure send to 2 nonblocking secure send to 1 nonblocking secure recv from 2 nonblocking secure revc from 1 blah blah blah wait for send to 2 wait for send to 1 wait for recv from 2 wait for recv from 1 is safe (by definition?). It will not deadlock with any correct implementation of MPI. Secondary proposal ------------------ MPI should not contain a combination of SECURE complete mode and READY start mode. I can see no sense in such a combination, and leaving it out decreases the overlap between the safe subset of MPI and the potentially unsafe subset of MPI. /--------------------------------------------------------\ e||) | Lyndon J Clarke Edinburgh Parallel Computing Centre | e||) c||c | Tel: 031 650 5021 Email: lyndon@epcc.edinburgh.ac.uk | c||c \--------------------------------------------------------/ From owner-mpi-pt2pt@CS.UTK.EDU Wed Mar 3 16:54:07 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA11917; Wed, 3 Mar 93 16:54:07 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA08053; Wed, 3 Mar 93 16:53:08 -0500 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Wed, 3 Mar 1993 16:53:07 EST Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from ssdintel.ssd.intel.com by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA08044; Wed, 3 Mar 93 16:53:05 -0500 Received: from ernie.ssd.intel.com by SSD.intel.com (4.1/SMI-4.1) id AA20435; Wed, 3 Mar 93 13:52:55 PST Message-Id: <9303032152.AA20435@SSD.intel.com> To: lyndon@epcc.ed.ac.uk Cc: mpi-pt2pt@cs.utk.edu, prp@SSD.intel.com Subject: Re: safe, unsafe and erroneous MPI programs In-Reply-To: Your message of "Wed, 03 Mar 93 20:25:31 GMT." <6861.9303032025@subnode.epcc.ed.ac.uk> Date: Wed, 03 Mar 93 13:52:31 -0800 From: prp@SSD.intel.com I agree that there should be a way to produce programs which are known to be safe. However, I also think that once a program is proven safe, it should be easy to allow it to take advantage of buffering when available. To make it easy, I recommend a global safeness mode, rather that one that must be selected on each send and receive. I would suggest that the environmental management section contain a call in which the user can specify that all operations are to have SECURE semantics if coded as normal semantics, and those coded with READY semantics would either be proved correct (might be hard) or flagged as erroneous. This way, a program can be proved safe and then released from SECURE semantics with a single line change, instead of going through all the sends and receives and changing them from SECURE to normal. This is particularly important if we decide to distinguish modes by syntax rather than by a mode parameter. (I think we should use syntax because it can be more efficient.) So, I would propose a call like this: MPI_safe() which, if present, will cause the implementation to flag an unsafe program as erroneous. Alternatively, it could be a link option (that is, the MPI standard would require that the implementor provide a means of specifying that a program is to be created or executed in such a way as to report an error if any unsafe operation is attempted.) This would be analogous to lint for C programs. You use it when you need it, and the rest of the time it is not in the way. Paul From owner-mpi-pt2pt@CS.UTK.EDU Thu Mar 4 06:57:23 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA13100; Thu, 4 Mar 93 06:57:23 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA19151; Thu, 4 Mar 93 06:55:09 -0500 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Thu, 4 Mar 1993 06:55:08 EST Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from daedalus.epcc.ed.ac.uk by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA19137; Thu, 4 Mar 93 06:55:05 -0500 Date: Thu, 4 Mar 93 11:55:00 GMT Message-Id: <7359.9303041155@subnode.epcc.ed.ac.uk> From: L J Clarke Subject: Re: safe, unsafe and erroneous MPI programs To: prp@SSD.intel.com In-Reply-To: prp@SSD.intel.com's message of Wed, 03 Mar 93 13:52:31 -0800 Reply-To: lyndon@epcc.ed.ac.uk Cc: mpi-pt2pt@cs.utk.edu I'm afraid that I must disagree with the notion of a global SECURE mode as Paul has suggested, which can be switched on or off as the programmer/user desires. In the first case, we really must think about programs being written to include compositions of parallel libraries which have already been written and determined to be correct and secure. So the library writer should have control of how the library communications are performed, and this kind of "global change the semantics of everything" should be avoided and discouraged. In the second case, once a module has correctly and safely been written using secure communications I really don't think there will be so much to "take advatage of" in use of system buffering. All of my colleagues at EPCC program in this way, and all of our library work is done in this way. Using the secure communications and blocking/nonblocking forms one can write, without difficulty, secure and efficient software. In fact, more efficiently, in principle, since less memory copies of message data are needed. Best Wishes Lyndon /--------------------------------------------------------\ e||) | Lyndon J Clarke Edinburgh Parallel Computing Centre | e||) c||c | Tel: 031 650 5021 Email: lyndon@epcc.edinburgh.ac.uk | c||c \--------------------------------------------------------/ From owner-mpi-pt2pt@CS.UTK.EDU Thu Mar 4 07:17:32 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA20778; Thu, 4 Mar 93 07:17:32 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA19895; Thu, 4 Mar 93 07:16:45 -0500 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Thu, 4 Mar 1993 07:16:43 EST Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from daedalus.epcc.ed.ac.uk by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA19885; Thu, 4 Mar 93 07:16:36 -0500 Date: Thu, 4 Mar 93 12:16:27 GMT Message-Id: <7397.9303041216@subnode.epcc.ed.ac.uk> From: L J Clarke Subject: Re: Task identifiers To: mpi-pt2pt@cs.utk.edu In-Reply-To: Tony Skjellum's message of Wed, 3 Mar 93 16:01:25 CST Reply-To: lyndon@epcc.ed.ac.uk > I think that we agreed to defer dynamic issues to a future MPI standard > level, but I do agree that this is never guaranteed to happen, as Rusty > has said. Unless of course we all want to and can afford from our funding more beano trips to Dallas ;-) (joking, honest guv!) > The lack of control for processes, and the lack of dynamic > group syntax/semantics worries me. They will almost certainly appear as > non-portable extensions. Yup, probably, over what timescale do we think? > Perhaps the original N-month limit was too short to permit the best > possible standard. > Perhaps, but on the other hand dynamic groups (i.e., those where the membership changes in time during the existence of the group, if I understand the emergent MPI jargon correctly (hey, we need a glossary of MPI terminology, any volunteers?)) really do seem to be at an experimental stage in our worlds. We have dynamic groups in the system we have implemented (CHIMP), but to be quite honest hardly any users have really used them to implement dynamic groups, rather they have implemented static groups (i.e., those where the membership does not change in time during the existence of the group, if I understand the emergent MPI jargon correctly (hey, we need a glossary of MPI terminology, any volunteers?)). Dynamic groups have been put into PVM3, as experimental functions which may or may not survive in the current or some other form. In short, it seems to be just too early to try to define a standard for dynamic groups (glossary), which is of course unfortunate. I'd suggest strongly that we must have an open mind toward - or even plan for - a later invocation of MPI, at which point subjects such as dynamic group and dynamic process models are dealt with. Best Wishes Lyndon /--------------------------------------------------------\ e||) | Lyndon J Clarke Edinburgh Parallel Computing Centre | e||) c||c | Tel: 031 650 5021 Email: lyndon@epcc.edinburgh.ac.uk | c||c \--------------------------------------------------------/ From owner-mpi-pt2pt@CS.UTK.EDU Thu Mar 4 09:48:51 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA28327; Thu, 4 Mar 93 09:48:51 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA27405; Thu, 4 Mar 93 09:47:40 -0500 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Thu, 4 Mar 1993 09:47:39 EST Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from gstws.EPM.ORNL.GOV by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA27396; Thu, 4 Mar 93 09:47:38 -0500 Received: by gstws.epm.ornl.gov (AIX 3.2/UCB 5.64/4.03) id AA17713; Thu, 4 Mar 1993 09:47:36 -0500 Date: Thu, 4 Mar 1993 09:47:36 -0500 From: geist@gstws.epm.ornl.gov (Al Geist) Message-Id: <9303041447.AA17713@gstws.epm.ornl.gov> To: mpi-pt2pt@cs.utk.edu Subject: So many proposals, so few standards... I just wanted to put in my endorsement for Bill and Rusty's proposal to have three levels. The lowest having Marc's init, start, wait, free, etc. constructs. The next level having send and receives using buffer descriptors. The highest level send and receives specify the buffer and the datatype and send only contiguous bytes. I also like the idea of having opaque task id. This has been brought up by Peter, Tony, and now Bill and Rusty. There are nice features that arise from this approach that makes computing in a dynamic, heterogeneous environment much easier and efficient. PVM has now gone to this method of task naming because of the advantages. There is still a need for users to think in terms of 0-[p-1] and this is logically provided by having a rank in a group. I also agree with Bill and Rusty that to write portable code we have to have a standard way to start up one or more tasks. The existing interfaces all provide some mechanism for doing this. Are we going to propose a standard that doesn't even provide some of the basic features of the exiting interfaces? With regard to opaque task id, spawning tasks, and dynamic groups, I see a double standard being applied that I do not like. When we discuss communication routines, many people vote for new totally untried (maybe unworkable) concepts because "they could lead to more efficient programs in the future." Fine. But when we talk about things like dynamic groups, people start saying "gosh, never been tried before. We better not do it." Some of our standard effort is future looking and other parts are backwards looking. I feel it is time to try to be consistent. One final thought. If we do not design an interface that is as flexible, intuitive, and easy to use as existing interfaces, then we will have simply created Yet Another Message Passing interface (YAMP) and not a standard. Al Geist From owner-mpi-pt2pt@CS.UTK.EDU Thu Mar 4 11:17:07 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA00544; Thu, 4 Mar 93 11:17:07 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA01967; Thu, 4 Mar 93 11:14:31 -0500 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Thu, 4 Mar 1993 11:14:29 EST Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from Aurora.CS.MsState.Edu by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA01959; Thu, 4 Mar 93 11:14:28 -0500 Received: by Aurora.CS.MsState.Edu (4.1/6.0s-FWP); id AA05686; Thu, 4 Mar 93 10:11:41 CST Date: Thu, 4 Mar 93 10:11:41 CST From: Tony Skjellum Message-Id: <9303041611.AA05686@Aurora.CS.MsState.Edu> To: mpi-pt2pt@cs.utk.edu, geist@gstws.epm.ornl.gov Subject: Re: So many proposals, so few standards... I concur with Al. - Tony From owner-mpi-pt2pt@CS.UTK.EDU Thu Mar 4 11:21:55 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA00629; Thu, 4 Mar 93 11:21:55 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA02226; Thu, 4 Mar 93 11:19:52 -0500 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Thu, 4 Mar 1993 11:19:50 EST Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from daedalus.epcc.ed.ac.uk by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA02173; Thu, 4 Mar 93 11:19:42 -0500 Date: Thu, 4 Mar 93 16:19:30 GMT Message-Id: <7649.9303041619@subnode.epcc.ed.ac.uk> From: L J Clarke Subject: Re: So many proposals, so few standards... To: geist@gstws.epm.ornl.gov (Al Geist), mpi-pt2pt@cs.utk.edu In-Reply-To: Al Geist's message of Thu, 4 Mar 1993 09:47:36 -0500 Reply-To: lyndon@epcc.ed.ac.uk Okay, this is mainly about task identifiers again. [stuff deleted] > I also like the idea of having opaque task id. This has been brought up > by Peter, Tony, and now Bill and Rusty. There are nice features that > arise from this approach that makes computing in a dynamic, heterogeneous > environment much easier and efficient. PVM has now gone to this > method of task naming because of the advantages. > There is still a need for users to think in terms of 0-[p-1] > and this is logically provided by having a rank in a group. if (MPI will provide process groups) then One thing processes do really find useful - from hard experience - is to be able to receive a message from any member of the process group. If we frame point-to-point in terms of some opaque (global) task identifier type then how can we provide the very expressive receive selection above? If we can't, then opaque types do not seem capable of providing expressive power for process groups. fi > I also agree with Bill and Rusty that to write portable code > we have to have a standard way to start up one or more tasks. > The existing interfaces all provide some mechanism for doing this. > Are we going to propose a standard that doesn't even provide > some of the basic features of the exiting interfaces? Either one includes the task creation in the source code of the program in which case MPI somehow ought to look at a mechanism compatible with the overall conceptual content of MPI Or one puts the task creation outside the program and outside the domain of MPI. This also seems to boil down to whether we have a dynamic process model or a static process model. When we decide this then we can look at the details, but I really do think MPI should agree which the goal is before trying to reach consensus on the attack plan. Why don't we email vote on this? My vote: static model. > With regard to opaque task id, spawning tasks, and dynamic groups, > I see a double standard being applied that I do not like. > When we discuss communication routines, many people vote for > new totally untried (maybe unworkable) concepts because > "they could lead to more efficient programs in the future." Can we find examples of this? Let me see off the top of my head ... a) Communication handles, unless I missed somthing. b) Ready sender communication (symmetry partner of ready receiver) c) Buffer descriptiors with data types > Fine. But when we talk about things like dynamic groups, > people start saying "gosh, never been tried before. We better not do it." > Some of our standard effort is future looking and other parts > are backwards looking. I feel it is time to try to be consistent. Unfortunately it is the burden of a standards committee that it should standardise only agreed existing practice, in such a manner that imminent developments are anticipated and a route for extension of the standard at such a time that such developments become common practice is practical, insofar as the standards committee is able to so do. My phrasing may not be the best, but the principle should be perfectly clear. > One final thought. If we do not design an interface that is > as flexible, intuitive, and easy to use as existing interfaces, > then we will have simply created Yet Another Message Passing interface (YAMP) > and not a standard. i.e., it will not be acceptable to a sufficiently large portion of the market and therefore will not be accepted by the market as a 'de facto' industry standard. How about some serious suggestions regarding how MPI can ensure to avoid this apocalypse? Best Wishes Lyndon /--------------------------------------------------------\ e||) | Lyndon J Clarke Edinburgh Parallel Computing Centre | e||) c||c | Tel: 031 650 5021 Email: lyndon@epcc.edinburgh.ac.uk | c||c \--------------------------------------------------------/ From owner-mpi-pt2pt@CS.UTK.EDU Thu Mar 4 12:53:07 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA03951; Thu, 4 Mar 93 12:53:07 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA07822; Thu, 4 Mar 93 12:50:08 -0500 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Thu, 4 Mar 1993 12:50:06 EST Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from gstws.EPM.ORNL.GOV by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA07811; Thu, 4 Mar 93 12:50:05 -0500 Received: by gstws.epm.ornl.gov (AIX 3.2/UCB 5.64/4.03) id AA17511; Thu, 4 Mar 1993 12:50:03 -0500 Date: Thu, 4 Mar 1993 12:50:03 -0500 From: geist@gstws.epm.ornl.gov (Al Geist) Message-Id: <9303041750.AA17511@gstws.epm.ornl.gov> To: mpi-pt2pt@cs.utk.edu Subject: Re: Reply to Lyndon >is to be able to receive a message from any member of the process group. >If we frame point-to-point in terms of some opaque (global) task >identifier type then how can we provide the very expressive >receive selection above? The receive side is easy, and if you REALLY mean process group above then the send side is also. But if you refuse to use the rank in a group then knowing who to send to is the trick. >Either one includes the task creation in the source code of the program >Or one puts the task creation outside the program... Putting it outside only works for multprocessors. What if you have multiple multiprocessors or just a bunch of workstations on a network? Users would have to write a separate (non-portable) program just to start up their applications. The argument still boils down to portability. >Why don't we email vote on this? My vote is dynamic model >Unfortunately it is the burden of a standards committee that it should >standardise only agreed existing practice, This is what I read as one of our original GOALS in MPI. But that is not what I see happening. We either need to stick to the original goal and drop all the "fringe" stuff out of the communication specification, or we take a consistently forward-looking approach (which can't possibly be agreed on in two more meetings) but may be the only way that we >can ensure to avoid this apocalypse? Al Geist From owner-mpi-pt2pt@CS.UTK.EDU Fri Mar 5 03:26:45 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA23608; Fri, 5 Mar 93 03:26:45 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA22531; Fri, 5 Mar 93 03:25:56 -0500 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Fri, 5 Mar 1993 03:25:55 EST Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from pnlg.pnl.gov by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA22521; Fri, 5 Mar 93 03:25:52 -0500 Received: from carbon.pnl.gov (130.20.188.38) by pnlg.pnl.gov; Thu, 4 Mar 93 13:22 PST Received: from sodium.pnl.gov by carbon.pnl.gov (4.1/SMI-4.1) id AA11676; Thu, 4 Mar 93 13:21:48 PST Received: by sodium.pnl.gov (4.1/SMI-4.0) id AA10514; Thu, 4 Mar 93 13:21:44 PST Date: Thu, 4 Mar 93 13:21:44 PST From: d39135@sodium.pnl.gov Subject: RE: Reply to Lyndon To: geist@gstws.epm.ornl.gov, mpi-pt2pt@cs.utk.edu Cc: d39135@sodium.pnl.gov Message-Id: <9303042121.AA10514@sodium.pnl.gov> X-Envelope-To: mpi-pt2pt@cs.utk.edu Al Geist writes: > Putting it outside only works for multprocessors. What if you have > multiple multiprocessors or just a bunch of workstations on a network? > Users would have to write a separate (non-portable) program just > to start up their applications. I am puzzled by these words. There is at least one package (TCGMSG, by Robert Harrison) that uses external invocation and works quite nicely on workstation networks as well as multiprocessors. In TCGMSG, the programming interface is that the application just calls PBEGIN_(argc,argv) to initialize, then finds out the number of participating processes by calling NNODES(). (This is a static process model.) For workstations, TCGMSG supplies a general purpose program to start up application processes. It works by passing them command-line flags that allow them to find each other. The application does not need to worry about these flags, because they are handled by the PBEGIN_ . On multiprocessors, other procedures may be used. For example, on the DELTA, the application processes are initiated by the Intel utility 'mexec', and PBEGIN_ works by calling the Intel intrinsic numnodes(). > The argument still boils down to portability. Exactly. As discussed above, the start-em-up-externally approach used by TCGMSG allows a program to execute without change across widely varying environments, with no need for the application programmer to write non-portable code. It would have been extremely difficult, verging on impossible, to support an interface that explicitly allows an application to create new processes on the fly. (That capability is not provided by the DELTA operating system. But they do support sockets, so I guess you could have a demon on some other system that would rsh back to the delta and do another mexec, then use some undocumented and risky system interfaces to pass messages between the nodes of the two mexec's, and then... But that doesn't seem very robust.) > >Why don't we email vote on this? I propose that MPI simply mandate the existence of a capability to start up processes externally. Those processes find themselves executing spontaneously and then find out about the others by making MPI calls. The syntax and semantics of the MPI calls should be specified completely, but we should not say anything about the external capability except that it exists. --Rik Littlefield From owner-mpi-pt2pt@CS.UTK.EDU Fri Mar 5 09:45:27 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA27927; Fri, 5 Mar 93 09:45:27 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA14009; Fri, 5 Mar 93 09:43:32 -0500 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Fri, 5 Mar 1993 09:43:30 EST Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from marge.meiko.com by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA14000; Fri, 5 Mar 93 09:43:26 -0500 Received: from hub.meiko.co.uk by marge.meiko.com with SMTP id AA22099 (5.65c/IDA-1.4.4 for ); Fri, 5 Mar 1993 09:43:22 -0500 Received: from float.co.uk (float.meiko.co.uk) by hub.meiko.co.uk (4.1/SMI-4.1) id AA05607; Fri, 5 Mar 93 14:43:17 GMT Date: Fri, 5 Mar 93 14:43:16 GMT From: jim@meiko.co.uk (James Cownie) Message-Id: <9303051443.AA05607@hub.meiko.co.uk> Received: by float.co.uk (5.0/SMI-SVR4) id AA02088; Fri, 5 Mar 93 14:40:29 GMT To: mpi-pt2pt@cs.utk.edu Subject: Cancel Content-Length: 4971 People, I believe that any of the non-blocking operations which MPI mandates should be cancellable. Why have cancel ? ================= There are various reasons for this (in random order) 1) Resource liberation Some (? ALL ?) of these operations commit system resources. If the user knows that the operation can never complete, then she should be able to release those resources. 2) Access revocation Some of these operations give the MPI system authority to modify areas of the user's store. (non blocking receive in particular). The user should be able to revoke this permission. [See examples with receive buffers on the stack for the true potential horror here !] 3) Cleanliness In some of Marc's early documents (and maybe still) there is an implication that all messages MUST have been received at program exit, otherwise the program is erroneous. It seems reasonable also to require all non-blocking operations to have completed (even MPI -1.1's non blocking exit call completes at this point !), if this is so then we need to be able to force the completion of these operations. (This is particularly the case for server type objects which may wish to queue many non-blocking receives [so as to avoid system buffering and additional data copying], but then wish to exit, possibly returning to the rest of the code...). What does cancel mean ? ======================= Cancelling an operation forces its completion with an error status of CANCELLED. One can reasonably require that the cancelled operation then be tested for completion in the normal way that any such non-blocking operation would be. Cancelling an operation can FAIL. This occurs if the operation to be cancelled has already successfully completed (Entered the state in which it could be successfully found as complete) or the system can guarantee its completion in a bounded time (in which case the cancel call should block until the operation completes). After an operation has been cancelled, I know that either 1) The operation completed normally or 2) The operation will never now proceed (it has been cancelled) The two cases can be separated by testing the completion return from the normal WAIT or STATUS call. (This is preferable to returning this result as the status of the cancel call as it makes the case of one thread cancelling a call on which another thread is already waiting resolve correctly). Consider for example cancelling a non-blocking send operation, there are three possible cases :- 1) data transfer has not actually started => it never starts, the operation is really cancelled, returning a CANCELLED kind of result. 2) data transfer has already completed => cancel failed and returns a COMPLETED kind of result 3) data transfer has started, so it is allowed to complete then reduces to case 2) Note that cancellation is an operation which happens to a REQUEST, it is not an operation which is applied to a MESSAGE. Therefore once a particular receive has been (successfully) cancelled the system state is as if that receive had never existed. There is NO implication that the cancel has an effect on any future incoming message which would have matched the, now cancelled, receive. Such a message can still be received in the normal way by any suitable matching receive call. Cancelling Group operations =========================== If we have non-blocking group operations we should be able to cancel them. It seems entirely reasonable to require that all the participants who have started the operation have also to cancel it. Issues ====== People are nervous about cancel because there is a race condition involved. This race is between the cancel operation itself, and the external event which signals completion of the operation being cancelled. (e.g. the cancel and the incoming message on a non-blocking receive). However this race is no worse than the existing one between a receive and an incoming message. (Does the data go straight into the user buffer, or require system buffering ??) Implementing a full cancel as described above is non-trivial, since it may be necessary to send out a cancel message after a non-blocking send. (This is the case if one uses receiver buffering). If this is viewed as too complex, (however note that various people already have implementations of cancel which implement semantics similar to those described above), then at the very least we should still seriously consider including the ability to cancel operations which gave the system permission to write into our address space. (Thanks to Lyndon for some of the examples and other discussion). -- Jim James Cownie Meiko Limited Meiko Inc. 650 Aztec West Reservoir Place Bristol BS12 4SD 1601 Trapelo Road England Waltham MA 02154 Phone : +44 454 616171 +1 617 890 7676 FAX : +44 454 618188 +1 617 890 5042 E-Mail: jim@meiko.co.uk or jim@meiko.com From owner-mpi-pt2pt@CS.UTK.EDU Fri Mar 5 10:05:14 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA28310; Fri, 5 Mar 93 10:05:14 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA15328; Fri, 5 Mar 93 10:04:29 -0500 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Fri, 5 Mar 1993 10:04:26 EST Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from Aurora.CS.MsState.Edu by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA15320; Fri, 5 Mar 93 10:04:25 -0500 Received: by Aurora.CS.MsState.Edu (4.1/6.0s-FWP); id AA06600; Fri, 5 Mar 93 09:01:26 CST Date: Fri, 5 Mar 93 09:01:26 CST From: Tony Skjellum Message-Id: <9303051501.AA06600@Aurora.CS.MsState.Edu> To: geist@gstws.epm.ornl.gov, mpi-pt2pt@cs.utk.edu, d39135@sodium.pnl.gov Subject: RE: Reply to Lyndon Rik, Using the Delta's lack of an elegant process model to defend your point is not really valid. They could have supported dynamic process creation, but just were too busy with other things, and did not see how useful the previously supported feature(s) were from iPSC960 and iPSC/1&2. The real issue is whether anything is lost/gained by separating spawning and MPI messaging. Why might there be? 1) ability to spawn processes in a way to take advantage of special group properties (hierarchy in the memory/network access). Oh well, the user can still do this him/herself, and then tell MPI that the group has a special property. 2) ability to support dynamic process model to allow MPI to serve as a base layer for additional services. For true portability, I would like to avoid having MEXEC on Delta, and spawn() functions on iPSC860 and CE/RK, and so on. The lack of a well-understood need for group, so that group-scope operations, can be attributed to a group, have not been recognized in MPI. As each group might want to implement a different broadcast or combine or other operations, the lack of a well-formed group concept is the bottleneck to this discussion. I am sleepy, I can't think of more now. - Tony From owner-mpi-pt2pt@CS.UTK.EDU Fri Mar 5 11:07:36 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA00459; Fri, 5 Mar 93 11:07:36 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA20433; Fri, 5 Mar 93 11:06:47 -0500 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Fri, 5 Mar 1993 11:06:45 EST Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from daedalus.epcc.ed.ac.uk by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA20425; Fri, 5 Mar 93 11:06:37 -0500 Date: Fri, 5 Mar 93 16:06:03 GMT Message-Id: <8555.9303051606@subnode.epcc.ed.ac.uk> From: L J Clarke Subject: Re: Cancel To: jim@meiko.co.uk (James Cownie), mpi-pt2pt@cs.utk.edu In-Reply-To: James Cownie's message of Fri, 5 Mar 93 14:43:16 GMT Reply-To: lyndon@epcc.ed.ac.uk Hello world :-) I just like to say that I generally support Jim's proposal. I see three sensible options, at this point: (1) Cancel for nonblocking receive operations (2) Cancel for nonblocking point-to-point operations (i.e., send and receive) (3) Cancel for nonblocking operations (i.e., point-to-point and collective) [How about cancellable nonblocking group partitions? :-)] I will certainly vote for (2) - I don't suppose there are any surprises there. I expect that I will find in favour of (3), but I really have to think through implementation first, so I'm reserving judgement. Best Wishes Lyndon /--------------------------------------------------------\ e||) | Lyndon J Clarke Edinburgh Parallel Computing Centre | e||) c||c | Tel: 031 650 5021 Email: lyndon@epcc.edinburgh.ac.uk | c||c \--------------------------------------------------------/ From owner-mpi-pt2pt@CS.UTK.E