From owner-mpi-comm@CS.UTK.EDU Mon Nov 23 16:08:35 1992 Return-Path: Received: by CS.UTK.EDU (5.61++/2.8s-UTK) id AA26326; Mon, 23 Nov 92 15:50:10 -0500 Received: from THUD.CS.UTK.EDU by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA26317; Mon, 23 Nov 92 15:50:05 -0500 From: Jack Dongarra Received: by thud.cs.utk.edu (5.61++/2.7c-UTK) id AA06935; Mon, 23 Nov 92 15:50:04 -0500 Date: Mon, 23 Nov 92 15:50:04 -0500 Message-Id: <9211232050.AA06935@thud.cs.utk.edu> To: mpi-comm@cs.utk.edu Subject: mpi committees and mailing list Status: R The following mailing lists have been set up. mpi-comm@cs.utk.edu Whole committee mpi-intro@cs.utk.edu Introduction subcommittee mpi-pt2pt@cs.utk.edu Point-to-point communication subcommittee mpi-collcomm@cs.utk.edu Collective communication subcommittee mpi-ptop@cs.utk.edu Process topology subcommittee mpi-lang@cs.utk.edu Language binding subcommittee mpi-formal@cs.utk.edu Formal language description subcommittee mpi-envir@cs.utk.edu Environment inquiry subcommittee All mail will be collected and can be retrieved by sending email to netlib@ornl.gov and in the mail message typing: send mpi-comm from mpi send mpi-intro from mpi etc. Also try: send index from mpi I'm in the process of collecting information on how the HPF committee operates and will pass along the details in the next few days. I would like to suggest January 6 - 8 for the next meeting in Dallas at the airport hotel. We should plan on starting around 1:00 on January 6th and finishing around 3:00 on January 8th. Below is a list of the subcommittees as I have them from our meeting last week. Please let me know if you have any changes. Regards, Jack Introduction Subcommittee ---------------------- Jack Dongarra - Chair dongarra@cs.utk.edu Daniel Frye danielf@kgnvma.vnet.ibm.com Tony Hey ajgh@ecs.soton.ac.uk Rusty Lusk lusk@mcs.anl.gov Steve Zenith zenith@kai.com Point-To-Point Communication Subcommittee ----------------------------------------- Marc Snir - Chair snir@watson.ibm.com Scott Berryman berryman@cs.yale.edu Jack Dongarra dongarra@cs.utk.edu Al Geist geist@msr.epm.ornl.gov Adam Greenberg moose@think.com Bill Gropp gropp@mcs.anl.gov Leslie Hart hart@fsl.noaa.gov Rolf Hempel hempel@gmd.de Tom Henderson hender@fsl.noaa.gov Bob Knighten knighten@ssd.intel.com Oliver McBryan mcbryan@piper.cs.colorado.edu Peter Rigsbee par@cray.com David Walker walker@msr.epm.ornl.gov Collective Communication Subcommittee ------------------------------------- Al Geist - Chair geist@msr.epm.ornl.gov Leslie Hart hart@fsl.noaa.gov Jon Flowers jwf@parasoft.com Adam Greenberg moose@think.com Bob Knighten knighten@ssd.intel.com Steve Otto otto@cse.ogi.edu Peter Rigsbee par@cray.com Marc Snir snir@watson.ibm.com David Walker walker@msr.epm.ornl.gov Process Topology Subcommittee ----------------------------- Rolf Hempel - Chair hempel@gmd.de Jon Flowers jwf@parasoft.com Tom Henderson hender@fsl.noaa.gov Oliver McBryan mcbryan@piper.cs.colorado.edu Steve Otto otto@cse.ogi.edu Lew Tucker tucker@think.com Language Binding Subcommittee ----------------------------- Scott Berryman - Chair berryman@cs.yale.edu Bruce Leisure bleasure@kai.com Formal Language Description Subcommittee ---------------------------------------- Steve Zenith - Chair zenith@kai.com Tony Hey ajgh@cs.ston.ac.uk Rusty Lusk lusk@mcs.anl.gov Environment Inquiry Subcommittee --------------------------------- Bill Gropp - Chair gropp@mcs.anl.gov Daniel Frye danielf@kgnvma.vnet.ibm.com From owner-mpi-comm@CS.UTK.EDU Tue Nov 24 15:37:43 1992 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA02830; Tue, 24 Nov 92 15:37:43 -0500 Received: by CS.UTK.EDU (5.61++/2.8s-UTK) id AA19627; Tue, 24 Nov 92 15:12:24 -0500 Received: from antares.mcs.anl.gov by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA19622; Tue, 24 Nov 92 15:12:21 -0500 Received: from godzilla.mcs.anl.gov by antares.mcs.anl.gov (4.1/SMI-GAR) id AA25860; Tue, 24 Nov 92 14:12:19 CST From: gropp@antares.mcs.anl.gov (William Gropp) Received: by godzilla.mcs.anl.gov (4.1/GeneV4) id AA11600; Tue, 24 Nov 92 14:12:17 CST Date: Tue, 24 Nov 92 14:12:17 CST Message-Id: <9211242012.AA11600@godzilla.mcs.anl.gov> To: mpi-comm@cs.utk.edu Subject: Test Implementation of the Draft Standard We have completed a first and rough draft of a test implementation. We have included (and run!) three sample programs written in MPI on a sun network and on an Intel ipsc/i860. It is available by anonymous ftp from info.mcs.anl.gov in pub/mpi . The README file there gives more information, including how to get, install, and run the examples. This is a rough draft; the file mpi.man.ps.Z contains a description of the implementations architecture and man pages for all of the routines. We will be updating this implementation with additional examples and fewer restrictions soon. Enjoy! Bill Gropp and Rusty Lusk From owner-mpi-comm@CS.UTK.EDU Tue Nov 24 17:07:49 1992 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA06915; Tue, 24 Nov 92 17:07:49 -0500 Received: by CS.UTK.EDU (5.61++/2.8s-UTK) id AA21596; Tue, 24 Nov 92 16:45:52 -0500 Received: from THUD.CS.UTK.EDU by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA21572; Tue, 24 Nov 92 16:45:38 -0500 From: Jack Dongarra Received: by thud.cs.utk.edu (5.61++/2.7c-UTK) id AA21258; Tue, 24 Nov 92 16:45:35 -0500 Date: Tue, 24 Nov 92 16:45:35 -0500 Message-Id: <9211242145.AA21258@thud.cs.utk.edu> To: mpi-comm@cs.utk.edu Subject: working procedures Chuck Koelbel has passed on some notes from the HPF experence. I think we will find them useful in carring out our meetings. Jack HPFF Working Procedures HPFF never had an explicit vote on adopting any operating procedures. We did, however, informally adopt the following rules. General Organization Most of the technical details were hammered out in subgroups, which made the base proposals. The main group would discuss these proposals and (usually) adopt them. In practice, lots of the technical discussions were done on email lists. Keeping those lists public was a big part of the openness of the forum. For a while, each subgroup brought its own proposal as handouts. Eventually, we ended up with a unified draft thanks to Dave Loveman. The intent (which we never lived up to) was that all proposals discussed at a meeting would be in that draft. Despite always having a loose proposal or two, the draft was extremely useful - among other things, it made editing a final report possible. Main Group / Plenary Session Matters For running discussions, we used loosely-enforced Robert's Rules of Order. Basically, we didn't stand on formalities unless the discussion was becoming unruly. Some of the more often-invoked rules: 1. Moving and voting on amendments before the main proposal. Basically, this kept the discussions focussed. 2. Motions coming out of committee (subgroup) are automatically seconded; others need a second from the floor. 3. Triply-nested amendments are not allowed. This keeps the confusion down. Generally, whoever was making the presentation ran the discussion, recognized comments from the floor, etc. Ken usually came in after the meat of the discussion to run the votes. Peer pressure was fairly effective at keeping people from filibustering (I think once there was a call to limit discussion). More common was lots of people wanting to make comments - this was handled by the presenters, usually by saying "OK, let's go clockwise around the table". Organizations were limited to 2 attendees each (not enforced, but no major problems). We passed around an attendence sheet at each meeting to keep track of who was there; because of various planning problems, I recommend using a pre-registration scheme ("send email to xxx@yyy if you plan to attend"). Organizations were asked to commit to having the same attendee at every meeting, and this was generally followed. It's very important to have this kind of continuity in attendees, else we would have spent too much time in remedial education. Each organization (school, company, lab) got one vote - note this was on an organization basis, not a person. We didn't have trouble with cheating on this rule - just make sure the representatives from an organization have agreed on who is voting. An organization was eligible to vote if it had attended 2 of the last 3 meetings, counting the current meeting (i.e. you could attend every other meeting and still vote; you could not vote at your first meeting). Obviously, we couldn't enforce this rule at the first two meetings. Accepting a section of the HPF language spec was a multi-step process. 1. Someone wrote a draft specification; often there were a couple drafts. Details of these were hashed out in the subgroup. 2. First Reading: The subgroup leader (or occasionally the draft author) presented the subgroup-approved draft to the main group. When there was controversy, it was usually pointed out. The main group discussed, suggested changes (sometimes), and held a series of "straw votes" on the proposal. All attendees were eligible to vote in straw votes, which were not binding on the subgroups. 3. More subgroup discussion, both electronic and in person at the next meeting, producing a revised proposal. 4. Second Reading: The subgroup leader presented the revised proposal to the main group. Sections that were substantially the same as the original (or an alternative presented at the first reading) were amended by motion and eventually voted on. Eligibility for votes was as explained above. Major additions (for example, when PURE functions were added to FORALL between meetings) were treated as first readings at this point. Often most of the proposal was accepted, and a few sections were sent back to subgroup for more work; these came back as second readings at the next meeting. 5. Once a section was accepted at second reading, it was "frozen" until the end of the HPFF process. Revisions were only allowed for clarity or when new information surfaced (like discovering that the draft was self-contradictory). This limitation was enforced as strictly as we could, to avoid backtracking. 6. At the end of the process (December), we promised to allow reconsideration of any feature. See also the discussion of the Journal of Development below. Subgroup Matters Subgroups met independently of the main body, usually the afternoon and evening of the day before. The leader of the subgroup ran these meetings using whatever style he or she was comfortable with. Most of us took votes from whoever showed up, and ran a pseud-democratic style. Also, there was a mailing list for each subgroup where most of the discussions went on. Each subgroup was devoted to one topic from the following list: Data distribution FORALL and other parallel statements Fortran 90, storage & sequence association, and the HPF subset Intrinsic functions Parallel I/O EXTRINSIC (nee LOCAL, nee FOREIGN) functions The groups met in parallel, which caused a little friction but was logstically unavoidable. When a subject straddled two groups (like intrinsic functions for enquiring about distributions), the subgroup leaders would talk to each other and decide who would handle it - this often led to minor anti-turf battles (also known as "after you" deadlock). Both groups would act as sanity checks on the results. When it came time to write the draft, each subgroup became a chapter. The subgroup leader was the editor (and usually major author) of the chapter, and was responsible for making sure the chapter reflected the decisions made in the subgroup and in committee. Logistical Matters Meetings were 2.5 days, starting Wednesday afternoon, in Dallas (there were exceptions to this for logistical reasons, but we would have prefered to keep them all on this schedule). A typical schedule was Wednesday 1:30-6:00 Subgroup meetings, about 3 going on in parallel (reserve 4 "breakout" rooms with the hotel, fo 10-20 people each) 6:00-7:30 Unofficial dinner break (usually the subgroup leaders ate together & planned the meeting agenda) 7:30-10:30 More subgroup meetings (subgroup leaders usually ended up staying up late to revise drafts) Thursday 9:00-12:00 Full group meeting (coffee breaks at natural breaks) 12:00-1:30 Lunch (provided) 1:30-6:00 Full group meeting (and coffee breaks) 6:00-8:00 Dinner (attendees pay, but hotel provided transport to area restaurant) 8:00-10:00 (Sometimes) Full group meeting (when no full meeting, subgroups usually met instead) Friday 9:00-12:00 Full group meeting (and coffee breaks) 12:00-1:30 (Sometimes) Lunch (provided if we thought the meeting would last long) 1:30-3:00 (Sometimes) Full group meeting We tried to finish as early as possible on Friday, to allow people to catch flights. Since we scheduled all the meetings early on, we set up a contract with the hotel. I didn't handle the details, but I think this saved us a little money and a lot of aggravation (talk to Theresa Chatman if you need more info). At any rate, this is a great thing for someone on your staff to set up. We financed the meetings primarily from a per-meeting fee of $95 ($75 if we didn't have lunch the second day) per attendee. Ken offered to foot the bill for academics to attend, based on the expectation of NSF funds. We're still waiting on those funds; highly recommend that you don't make such offers without first getting the cash. Documentation The Draft: The earlier you have one the better. David Loveman served as our general editor, collecting the chapters and trying to smooth format details. I contributed the introduction and credits, mainly cribbed from meeting notes and other announcements. Guy Steele contributed a set of macros for BNF syntax and reproducing code segments. We tried to find some formatting from an ANSI standard, but no dice. Toward the end of the process, version control became a problem. We (tried to) set up the following framework: 1. David had the "official" version of the draft. He set deadlines for receiving the chapters, and edited for formatting. 2. David sent me the whole document when all the chapters were done. I LaTeXed it to check for system dependences (never a problem) 3. I sent the document to the "core" mail group (see below), put it out for anonymous FTP, and announced it on the net. 4. Any further changes were supposed to be made to David's edited copy, not the original version (this to make the next round of edits easier). The system worked pretty well, until the final draft when the wrong version of Chapter 6 got distributed... Each subgroup wrote one chapter, generally written and/or edited by the subgroup leader. While the meetings were going on, the draft contained all proposals still under consideration (Guy provided a "\alternative" macro that marked sections as being alternatives to each other). Also, the author of each section was identified in a footnote at the beginning of the section, along with the date (and occasionally other version information). In the current draft, we've moved summaries of some discarded proposals to the "Journal of Development" (primarily the ones dropped for lack of time). All proposals exist in the archives; we may put together a "rationale" document to explain some of the more controversial decisions. We're also planning to move the authorship information from footnotes to the credits chapter in the December draft. It's too early to know how this organization will work for the reader, but the modularity has helped in writing the document. Other lessons from the draft: 1. Make sure the chapter authors realize they are writing a draft chapter, not a stand-alone document. 2. Make sure the person in charge of the credits chapter has plenty of time to work on it (I spent 50% of my time for a week appeasing various corporate egos, wording things right, and checking spelling of names). 3. Proofread the document before sending it out - both spell checking and careful chapter reading. Mailing lists: Every subgroup had its own mailing list. Those lists are where most of the technical action happened; they were very high-volume (final total was something like 1.5Mb of archived messages). On top of that, there was a list for everybody in the world interested in HPF, and another for the "core" group. The "everybody" list was used for the meeting minutes and a few miscellaneous announcements. The core group list was primarily for the meeting attendees, but a few others were also on it for political and/or practical reasons (for example, Gil Weigand and Theresa Chatman were both on the list); it got meeting details, and copies of the various proposals before the meetings. All lists were kept at cs.rice.edu (there were a couple smaller lists other places, including a European list at GMD and several local lists for particular campuses). People could add or delete themselves to/from any of the lists by mailing to another mail alias at Rice. Automating this was a huge win (although I got 2 or 3 requests sent directly to me every week). FTP: We made everything we could available by anonymous FTP at Rice; eventually other sites picked up some of it as well. This included the various drafts of the language spec; any other handouts from the meetings; supporting (sometimes critical) commentary; papers on Fortran D, Vienna Fortran, ADAPT, and other systems; archived email traffic; and the minutes of the meetings. Generally, I put up a couple versions of any document (particularly text formatting stuff) - TeX or LaTeX, Postscript, and compressed versions of large files. This was important, since I've found out some fascinating things about portability (or lack thereof) of different formats. (Hints: keep all lines 80 characters or less; no fancy macros in TeX; at most one "." in a file name (filename.tex.Z doesn't translate well to IBM!); never include ANY pathname.) Minutes: Extremely popular, but lots of work. I'm convinced that making them available was a key to HPFF's success. Make them as detailed as possible, particularly including counts of the votes taken, major topics discussed, and lists of people (humor helps occasionally, too). "Executive Summary" sections with the major issues at the front seemed to be popular. I also used the minutes to announce additions to the FTP archives. Advertizing: Announcements of major news (new drafts, etc.) went everywhere I could think of. This includes the "world" mailing list; newsgroups comp.lang.fortran, comp.lang.misc, and comp.parallel; (indirectly) na-net, scinet, and hpcwire. Meeting minutes went to the world and core mailing lists and the newsgroups. I've been writing short "What is HPF" articles constantly, and I think the same is true of others. Ken, Dave Loveman, Guy, and I have all given large numbers of talks on HPF; again, the same is probably true of others. This includes invited talks at conferences and visits to various companies. From chk@cs.rice.edu Mon Nov 23 15:45:48 1992 Return-Path: Received: from cs.rice.edu by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA26184; Mon, 23 Nov 92 15:45:39 -0500 Received: from DialupEudora (charon.rice.edu) by cs.rice.edu (AA22750); Mon, 23 Nov 92 14:44:21 CST Message-Id: <9211232044.AA22750@cs.rice.edu> Date: Mon, 23 Nov 1992 14:47:55 -0600 To: dongarra@cs.utk.edu From: chk@cs.rice.edu Subject: HPFF organization X-Attachments: :Macintosh HD:4227:HPFF rules: Status: RO Here it is. If anything is unclear, just ask me or (for logistical details) Theresa Chatman (tlc@cs.rice.edu). Chuck HPFF Working Procedures HPFF never had an explicit vote on adopting any operating procedures. We did, however, informally adopt the following rules. General Organization Most of the technical details were hammered out in subgroups, which made the base proposals. The main group would discuss these proposals and (usually) adopt them. In practice, lots of the technical discussions were done on email lists. Keeping those lists public was a big part of the openness of the forum. For a while, each subgroup brought its own proposal as handouts. Eventually, we ended up with a unified draft thanks to Dave Loveman. The intent (which we never lived up to) was that all proposals discussed at a meeting would be in that draft. Despite always having a loose proposal or two, the draft was extremely useful - among other things, it made editing a final report possible. Main Group / Plenary Session Matters For running discussions, we used loosely-enforced Robert's Rules of Order. Basically, we didn't stand on formalities unless the discussion was becoming unruly. Some of the more often-invoked rules: 1. Moving and voting on amendments before the main proposal. Basically, this kept the discussions focussed. 2. Motions coming out of committee (subgroup) are automatically seconded; others need a second from the floor. 3. Triply-nested amendments are not allowed. This keeps the confusion down. Generally, whoever was making the presentation ran the discussion, recognized comments from the floor, etc. Ken usually came in after the meat of the discussion to run the votes. Peer pressure was fairly effective at keeping people from filibustering (I think once there was a call to limit discussion). More common was lots of people wanting to make comments - this was handled by the presenters, usually by saying "OK, let's go clockwise around the table". Organizations were limited to 2 attendees each (not enforced, but no major problems). We passed around an attendence sheet at each meeting to keep track of who was there; because of various planning problems, I recommend using a pre-registration scheme ("send email to xxx@yyy if you plan to attend"). Organizations were asked to commit to having the same attendee at every meeting, and this was generally followed. It's very important to have this kind of continuity in attendees, else we would have spent too much time in remedial education. Each organization (school, company, lab) got one vote - note this was on an organization basis, not a person. We didn't have trouble with cheating on this rule - just make sure the representatives from an organization have agreed on who is voting. An organization was eligible to vote if it had attended 2 of the last 3 meetings, counting the current meeting (i.e. you could attend every other meeting and still vote; you could not vote at your first meeting). Obviously, we couldn't enforce this rule at the first two meetings. Accepting a section of the HPF language spec was a multi-step process. 1. Someone wrote a draft specification; often there were a couple drafts. Details of these were hashed out in the subgroup. 2. First Reading: The subgroup leader (or occasionally the draft author) presented the subgroup-approved draft to the main group. When there was controversy, it was usually pointed out. The main group discussed, suggested changes (sometimes), and held a series of "straw votes" on the proposal. All attendees were eligible to vote in straw votes, which were not binding on the subgroups. 3. More subgroup discussion, both electronic and in person at the next meeting, producing a revised proposal. 4. Second Reading: The subgroup leader presented the revised proposal to the main group. Sections that were substantially the same as the original (or an alternative presented at the first reading) were amended by motion and eventually voted on. Eligibility for votes was as explained above. Major additions (for example, when PURE functions were added to FORALL between meetings) were treated as first readings at this point. Often most of the proposal was accepted, and a few sections were sent back to subgroup for more work; these came back as second readings at the next meeting. 5. Once a section was accepted at second reading, it was "frozen" until the end of the HPFF process. Revisions were only allowed for clarity or when new information surfaced (like discovering that the draft was self-contradictory). This limitation was enforced as strictly as we could, to avoid backtracking. 6. At the end of the process (December), we promised to allow reconsideration of any feature. See also the discussion of the Journal of Development below. Subgroup Matters Subgroups met independently of the main body, usually the afternoon and evening of the day before. The leader of the subgroup ran these meetings using whatever style he or she was comfortable with. Most of us took votes from whoever showed up, and ran a pseud-democratic style. Also, there was a mailing list for each subgroup where most of the discussions went on. Each subgroup was devoted to one topic from the following list: Data distribution FORALL and other parallel statements Fortran 90, storage & sequence association, and the HPF subset Intrinsic functions Parallel I/O EXTRINSIC (nee LOCAL, nee FOREIGN) functions The groups met in parallel, which caused a little friction but was logstically unavoidable. When a subject straddled two groups (like intrinsic functions for enquiring about distributions), the subgroup leaders would talk to each other and decide who would handle it - this often led to minor anti-turf battles (also known as "after you" deadlock). Both groups would act as sanity checks on the results. When it came time to write the draft, each subgroup became a chapter. The subgroup leader was the editor (and usually major author) of the chapter, and was responsible for making sure the chapter reflected the decisions made in the subgroup and in committee. Logistical Matters Meetings were 2.5 days, starting Wednesday afternoon, in Dallas (there were exceptions to this for logistical reasons, but we would have prefered to keep them all on this schedule). A typical schedule was Wednesday 1:30-6:00 Subgroup meetings, about 3 going on in parallel (reserve 4 "breakout" rooms with the hotel, fo 10-20 people each) 6:00-7:30 Unofficial dinner break (usually the subgroup leaders ate together & planned the meeting agenda) 7:30-10:30 More subgroup meetings (subgroup leaders usually ended up staying up late to revise drafts) Thursday 9:00-12:00 Full group meeting (coffee breaks at natural breaks) 12:00-1:30 Lunch (provided) 1:30-6:00 Full group meeting (and coffee breaks) 6:00-8:00 Dinner (attendees pay, but hotel provided transport to area restaurant) 8:00-10:00 (Sometimes) Full group meeting (when no full meeting, subgroups usually met instead) Friday 9:00-12:00 Full group meeting (and coffee breaks) 12:00-1:30 (Sometimes) Lunch (provided if we thought the meeting would last long) 1:30-3:00 (Sometimes) Full group meeting We tried to finish as early as possible on Friday, to allow people to catch flights. Since we scheduled all the meetings early on, we set up a contract with the hotel. I didn't handle the details, but I think this saved us a little money and a lot of aggravation (talk to Theresa Chatman if you need more info). At any rate, this is a great thing for someone on your staff to set up. We financed the meetings primarily from a per-meeting fee of $95 ($75 if we didn't have lunch the second day) per attendee. Ken offered to foot the bill for academics to attend, based on the expectation of NSF funds. We're still waiting on those funds; highly recommend that you don't make such offers without first getting the cash. Documentation The Draft: The earlier you have one the better. David Loveman served as our general editor, collecting the chapters and trying to smooth format details. I contributed the introduction and credits, mainly cribbed from meeting notes and other announcements. Guy Steele contributed a set of macros for BNF syntax and reproducing code segments. We tried to find some formatting from an ANSI standard, but no dice. Toward the end of the process, version control became a problem. We (tried to) set up the following framework: 1. David had the "official" version of the draft. He set deadlines for receiving the chapters, and edited for formatting. 2. David sent me the whole document when all the chapters were done. I LaTeXed it to check for system dependences (never a problem) 3. I sent the document to the "core" mail group (see below), put it out for anonymous FTP, and announced it on the net. 4. Any further changes were supposed to be made to David's edited copy, not the original version (this to make the next round of edits easier). The system worked pretty well, until the final draft when the wrong version of Chapter 6 got distributed... Each subgroup wrote one chapter, generally written and/or edited by the subgroup leader. While the meetings were going on, the draft contained all proposals still under consideration (Guy provided a "\alternative" macro that marked sections as being alternatives to each other). Also, the author of each section was identified in a footnote at the beginning of the section, along with the date (and occasionally other version information). In the current draft, we've moved summaries of some discarded proposals to the "Journal of Development" (primarily the ones dropped for lack of time). All proposals exist in the archives; we may put together a "rationale" document to explain some of the more controversial decisions. We're also planning to move the authorship information from footnotes to the credits chapter in the December draft. It's too early to know how this organization will work for the reader, but the modularity has helped in writing the document. Other lessons from the draft: 1. Make sure the chapter authors realize they are writing a draft chapter, not a stand-alone document. 2. Make sure the person in charge of the credits chapter has plenty of time to work on it (I spent 50% of my time for a week appeasing various corporate egos, wording things right, and checking spelling of names). 3. Proofread the document before sending it out - both spell checking and careful chapter reading. Mailing lists: Every subgroup had its own mailing list. Those lists are where most of the technical action happened; they were very high-volume (final total was something like 1.5Mb of archived messages). On top of that, there was a list for everybody in the world interested in HPF, and another for the "core" group. The "everybody" list was used for the meeting minutes and a few miscellaneous announcements. The core group list was primarily for the meeting attendees, but a few others were also on it for political and/or practical reasons (for example, Gil Weigand and Theresa Chatman were both on the list); it got meeting details, and copies of the various proposals before the meetings. All lists were kept at cs.rice.edu (there were a couple smaller lists other places, including a European list at GMD and several local lists for particular campuses). People could add or delete themselves to/from any of the lists by mailing to another mail alias at Rice. Automating this was a huge win (although I got 2 or 3 requests sent directly to me every week). FTP: We made everything we could available by anonymous FTP at Rice; eventually other sites picked up some of it as well. This included the various drafts of the language spec; any other handouts from the meetings; supporting (sometimes critical) commentary; papers on Fortran D, Vienna Fortran, ADAPT, and other systems; archived email traffic; and the minutes of the meetings. Generally, I put up a couple versions of any document (particularly text formatting stuff) - TeX or LaTeX, Postscript, and compressed versions of large files. This was important, since I've found out some fascinating things about portability (or lack thereof) of different formats. (Hints: keep all lines 80 characters or less; no fancy macros in TeX; at most one "." in a file name (filename.tex.Z doesn't translate well to IBM!); never include ANY pathname.) Minutes: Extremely popular, but lots of work. I'm convinced that making them available was a key to HPFF's success. Make them as detailed as possible, particularly including counts of the votes taken, major topics discussed, and lists of people (humor helps occasionally, too). "Executive Summary" sections with the major issues at the front seemed to be popular. I also used the minutes to announce additions to the FTP archives. Advertizing: Announcements of major news (new drafts, etc.) went everywhere I could think of. This includes the "world" mailing list; newsgroups comp.lang.fortran, comp.lang.misc, and comp.parallel; (indirectly) na-net, scinet, and hpcwire. Meeting minutes went to the world and core mailing lists and the newsgroups. I've been writing short "What is HPF" articles constantly, and I think the same is true of others. Ken, Dave Loveman, Guy, and I have all given large numbers of talks on HPF; again, the same is probably true of others. This includes invited talks at conferences and visits to various companies. From owner-mpi-comm@CS.UTK.EDU Tue Nov 24 18:09:18 1992 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA08323; Tue, 24 Nov 92 18:09:18 -0500 Received: by CS.UTK.EDU (5.61++/2.8s-UTK) id AA22602; Tue, 24 Nov 92 17:43:42 -0500 Received: from relay2.UU.NET by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA22596; Tue, 24 Nov 92 17:43:35 -0500 Received: from uunet.uu.net (via LOCALHOST.UU.NET) by relay2.UU.NET with SMTP (5.61/UUNET-internet-primary) id AA11950; Tue, 24 Nov 92 17:43:40 -0500 Received: from kailand.UUCP by uunet.uu.net with UUCP/RMAIL (queueing-rmail) id 174208.17287; Tue, 24 Nov 1992 17:42:08 EST Received: from brisk.kai.com (brisk) by kailand.kai.com via SMTP (5.65d-92031301) id AA07130; Tue, 24 Nov 1992 16:14:37 -0600 Received: by brisk.kai.com (920330.SGI-92101201) id AA06815; Tue, 24 Nov 92 16:14:36 -0600 Date: Tue, 24 Nov 92 16:14:36 -0600 Message-Id: <9211242214.AA06815@brisk.kai.com> To: dongarra@cs.utk.edu Cc: mpi-comm@cs.utk.edu In-Reply-To: Jack Dongarra's message of Tue, 24 Nov 92 15:04:05 -0500 <9211242004.AA20345@thud.cs.utk.edu> Subject: more on the intro From: Steven Ericsson Zenith Sender: zenith@kai.com Organization: Kuck and Associates, Inc. 1906 Fox Drive, Champaign IL USA 61820-7334, voice 217-356-2288, fax 217-356-5199 Concerning the introduction: "The paradigm will not be made obsolete by architectures combining the shared- and distributed-memory views, or by increases in network speeds." I think this sentence is unnecessarily defensive. I'd delete the sentence and the following "thus". I earlier raised an objection to \item A simple way to create processes for the SPMD model there seems to be some redundancy here with \item Process groups The former is a subset of the latter. I'll repeat my noncirculated comments: I would rather not see us say too much about the process model at this stage; except to say that there is some process model defined by the language using the standard and to say something about side effects between processes that affect the message passing semantics (i.e., that there should be none). That does, of course, leave us with some interesting problems for identifying our communication; traditionally a difficult subject in message passing (hence my distribution of this message to the whole committee). The easy route would be to have a global name space of shared "channels" each with a communication characteristic [type] (e.g., "point-to-point" or "one-to-many"): a compiler can check correct usage if we identify the right rules (e.g. a synchronized unidirectional point-to-point communication is shared by only two processes). Such logical naming would map well across all the existing systems and avoid the problems of name servers, processor id, topologies and so forth since any topology can be described by the set of names and its users. Processor ids are, in effect, names in a global name space - and are often channels that have a "many-to-one" characteristic. Steven From owner-mpi-comm@CS.UTK.EDU Sat Nov 28 18:38:03 1992 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA07921; Sat, 28 Nov 92 18:38:03 -0500 Received: by CS.UTK.EDU (5.61++/2.8s-UTK) id AA22407; Sat, 28 Nov 92 18:35:38 -0500 Received: from cunyvm.cuny.edu by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA22403; Sat, 28 Nov 92 18:35:36 -0500 Message-Id: <9211282335.AA22403@CS.UTK.EDU> Received: from YKTVMV by CUNYVM.CUNY.EDU (IBM VM SMTP V2R2) with BSMTP id 6064; Sat, 28 Nov 92 18:35:01 EST Date: Sat, 28 Nov 92 18:32:55 EST From: "Marc Snir" X-Addr: (914) 945-3204 (862-3204) 28-226 IBM T.J. Watson Research Center P.O. Box 218 Yorktown Heights NY 10598 To: mpi-comm@cs.utk.edu Subject: previously sent postcript file Reply-To: SNIR@watson.ibm.com Contains an outline I want to use for discussing issues in the poin-to-point subcommittee. I send it to the entire committee since it touches many issues related to collective communication and formal definitions. If somebody cannot print postscript and prefers latex, pls let me know. From owner-mpi-comm@CS.UTK.EDU Sat Nov 28 18:40:52 1992 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA07938; Sat, 28 Nov 92 18:40:52 -0500 Received: by CS.UTK.EDU (5.61++/2.8s-UTK) id AA22399; Sat, 28 Nov 92 18:33:55 -0500 Received: from cunyvm.cuny.edu by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA22394; Sat, 28 Nov 92 18:33:40 -0500 Message-Id: <9211282333.AA22394@CS.UTK.EDU> Received: from YKTVMV by CUNYVM.CUNY.EDU (IBM VM SMTP V2R2) with BSMTP id 6053; Sat, 28 Nov 92 18:33:04 EST Date: Sat, 28 Nov 92 18:32:22 EST From: "Marc Snir" To: MPI-COMM@CS.UTK.EDU %!PS-Adobe-2.0 %%Creator: dvips 5.47 Copyright 1986-91 Radical Eye Software %%Title: MPIOUTLN.DVI.* %%Pages: 10 1 %%BoundingBox: 0 0 612 792 %%EndComments %%BeginProcSet: texc.pro /TeXDict 250 dict def TeXDict begin /N /def load def /B{bind def}N /S /exch load def /X{S N}B /TR /translate load N /isls false N /vsize 10 N /@rigin{ isls{[0 1 -1 0 0 0]concat}if 72 Resolution div 72 VResolution div neg scale Resolution VResolution vsize neg mul TR matrix currentmatrix dup dup 4 get round 4 exch put dup dup 5 get round 5 exch put setmatrix}N /@letter{/vsize 10 N}B /@landscape{/isls true N /vsize -1 N}B /@a4{/vsize 10.6929133858 N}B /@a3{ /vsize 15.5531 N}B /@ledger{/vsize 16 N}B /@legal{/vsize 13 N}B /@manualfeed{ statusdict /manualfeed true put}B /@copies{/#copies X}B /FMat[1 0 0 -1 0 0]N /FBB[0 0 0 0]N /nn 0 N /IE 0 N /ctr 0 N /df-tail{/nn 8 dict N nn begin /FontType 3 N /FontMatrix fntrx N /FontBBox FBB N string /base X array /BitMaps X /BuildChar{CharBuilder} N /Encoding IE N end dup{/foo setfont}2 array copy cvx N load 0 nn put /ctr 0 N[}B /df{/sf 1 N /fntrx FMat N df-tail} B /dfs{div /sf X /fntrx[sf 0 0 sf neg 0 0]N df-tail}B /E{pop nn dup definefont setfont}B /ch-width{ch-data dup length 5 sub get} B /ch-height{ch-data dup length 4 sub get} B /ch-xoff{128 ch-data dup length 3 sub get sub} B /ch-yoff{ ch-data dup length 2 sub get 127 sub} B /ch-dx{ch-data dup length 1 sub get} B /ch-image{ch-data dup type /stringtype ne{ctr get /ctr ctr 1 add N}if}B /id 0 N /rw 0 N /rc 0 N /gp 0 N /cp 0 N /G 0 N /sf 0 N /CharBuilder{save 3 1 roll S dup /base get 2 index get S /BitMaps get S get /ch-data X pop /ctr 0 N ch-dx 0 ch-xoff ch-yoff ch-height sub ch-xoff ch-width add ch-yoff setcachedevice ch-width ch-height true[1 0 0 -1 -.1 ch-xoff sub ch-yoff .1 add]/id ch-image N /rw ch-width 7 add 8 idiv string N /rc 0 N /gp 0 N /cp 0 N{rc 0 ne{rc 1 sub /rc X rw}{G}ifelse}imagemask restore}B /G{{id gp get /gp gp 1 add N dup 18 mod S 18 idiv pl S get exec}loop}B /adv{cp add /cp X}B /chg{rw cp id gp 4 index getinterval putinterval dup gp add /gp X adv}B /nd{/cp 0 N rw exit}B /lsh{rw cp 2 copy get dup 0 eq{pop 1}{dup 255 eq{pop 254}{dup dup add 255 and S 1 and or}ifelse}ifelse put 1 adv}B /rsh{rw cp 2 copy get dup 0 eq{pop 128}{dup 255 eq{pop 127}{dup 2 idiv S 128 and or}ifelse}ifelse put 1 adv}B /clr{rw cp 2 index string putinterval adv}B /set{rw cp fillstr 0 4 index getinterval putinterval adv}B /fillstr 18 string 0 1 17{2 copy 255 put pop}for N /pl[{adv 1 chg}bind{adv 1 chg nd}bind{1 add chg}bind{1 add chg nd}bind{adv lsh}bind{ adv lsh nd}bind{adv rsh}bind{adv rsh nd}bind{1 add adv}bind{/rc X nd}bind{1 add set}bind{1 add clr}bind{adv 2 chg}bind{adv 2 chg nd}bind{pop nd}bind]N /D{ /cc X dup type /stringtype ne{]}if nn /base get cc ctr put nn /BitMaps get S ctr S sf 1 ne{dup dup length 1 sub dup 2 index S get sf div put}if put /ctr ctr 1 add N}B /I{cc 1 add D}B /bop{userdict /bop-hook known{bop-hook}if /SI save N @rigin 0 0 moveto}N /eop{clear SI restore showpage userdict /eop-hook known{eop-hook}if}N /@start{userdict /start-hook known{start-hook}if /VResolution X /Resolution X 1000 div /DVImag X /IE 256 array N 0 1 255{IE S 1 string dup 0 3 index put cvn put} for}N /p /show load N /RMat[1 0 0 -1 0 0]N /BDot 260 string N /rulex 0 N /ruley 0 N /v{/ruley X /rulex X V}B /V statusdict begin /product where{pop product dup length 7 ge{0 7 getinterval (Display)eq}{pop false}ifelse}{false}ifelse end{{gsave TR -.1 -.1 TR 1 1 scale rulex ruley false RMat{BDot}imagemask grestore}}{{gsave TR -.1 -.1 TR rulex ruley scale 1 1 false RMat{BDot}imagemask grestore}}ifelse B /a{moveto}B /delta 0 N /tail{dup /delta X 0 rmoveto}B /M{S p delta add tail}B /b{S p tail} B /c{-4 M}B /d{-3 M}B /e{-2 M}B /f{-1 M}B /g{0 M}B /h{1 M}B /i{2 M}B /j{3 M}B /k{4 M}B /w{0 rmoveto}B /l{p -4 w}B /m{p -3 w}B /n{p -2 w}B /o{p -1 w}B /q{p 1 w}B /r{p 2 w}B /s{p 3 w}B /t{p 4 w}B /x{0 S rmoveto}B /y{3 2 roll p a}B /bos{ /SS save N}B /eos{clear SS restore}B end %%EndProcSet TeXDict begin 1000 300 300 @start /Fa 1 16 df15 D E /Fb 51 125 df<127012F8B012701200A5 127012F8A31270051C779B18>33 D<137013F01201EA03C0EA0780EA0F00121E121C123C123812 781270A212F05AA87E1270A212781238123C121C121E7EEA0780EA03C0EA01F0120013700C2479 9F18>40 D<126012F012787E7E7EEA0780120313C0120113E01200A213F01370A813F013E0A212 0113C0120313801207EA0F00121E5A5A5A12600C247C9F18>I<123C127E127FA3123F120F120E 121E127C12F81270080C788518>44 DI48 DII<387FFFC0B512E0A3C8FCA4B512E0A36C13C0130C7E9318>61 D<137013F8A213D8A2EA01DCA3138CEA038EA41306EA0707A4380FFF80A3EA0E03A2381C01C0A2 387F07F038FF8FF8387F07F0151C7F9B18>65 DI<3801FCE0EA03FEEA07FFEA0F07EA1E03EA3C01EA78001270A200F013005AA87E0070 13E0A21278EA3C01001E13C0EA0F073807FF806C1300EA01FC131C7E9B18>IIII<3801F9C0EA07FF5AEA1F0FEA1C03123CEA78011270A200F0C7FC5AA5 EB0FF0131F130F38F001C0127013031278123CEA1C07EA1F0FEA0FFFEA07FDEA01F9141C7E9B18 >I<387F07F038FF8FF8387F07F0381C01C0A9EA1FFFA3EA1C01AA387F07F038FF8FF8387F07F0 151C7F9B18>II76 D<38FC01F8EAFE03A2383B06E0 A4138EA2EA398CA213DCA3EA38D8A213F81370A21300A638FE03F8A3151C7F9B18>I<387E07F0 38FF0FF8387F07F0381D81C0A313C1121CA213E1A313611371A213311339A31319A2131D130DA3 EA7F07EAFF87EA7F03151C7F9B18>III82 D<3807F380EA1FFF5AEA7C1FEA7007EAF00312E0A2 90C7FC7E1278123FEA1FF0EA0FFEEA01FF38001F80EB03C0EB01E01300A2126012E0130100F013 C0EAFC07B512801400EAE7FC131C7E9B18>I<387FFFF8B5FCA238E07038A400001300B2EA07FF A3151C7F9B18>I<38FF83FEA3381C0070B2001E13F0000E13E0EA0F013807C7C03803FF806C13 00EA007C171C809B18>I<38FF07F8A3381C01C0A4380E0380A4EA0F0700071300A4EA038EA4EA 018C13DCA3EA00D813F8A21370151C7F9B18>I<38FE03F8A338700070A36C13E0A513F8A2EA39 DCA2001913C0A3138CEA1D8DA4000D13801305EA0F07A2EA0E03151C7F9B18>I97 D<127E12FE127E120EA5133EEBFF80000F13C0EBE3E0EB80F0EB0070 1478000E1338A5120F14781470EB80F0EBC3E0EBFFC0000E138038067E00151C809B18>IIIII<3803F1F03807FFF85A381E1F30 383C0F00EA3807A5EA3C0FEA1E1EEA1FFC485AEA3BF00038C7FC123CEA1FFF14C04813E0387801 F038F00078481338A36C1378007813F0EA7E03383FFFE0000F13803803FE00151F7F9318>I<12 7E12FE127E120EA5133FEBFF80000F13C0EBE1E013801300A2120EAA387FC3FC38FFE7FE387FC3 FC171C809B18>II<12FEA3120EA5EB3FF0137F133FEB0780EB0F00131E5B5B5BEA0FF87F139C131EEA0E0FEB 0780130314C038FFC7F8A3151C7F9B18>107 DI<387DF1F038FFFBF86CB47E381F1F1CEA1E1EA2EA1C1CAB387F1F1F39FFBF BF80397F1F1F001914819318>IIII<387F87E038FF9FF8EA7FBF3803 FC78EBF030EBE0005BA35BA8EA7FFEB5FC6C5A15147F9318>114 DI<487E1203A4387FFFC0B5FCA238038000A9144014E0A21381EBC3C0EA01 FF6C1380EB7E0013197F9818>I<387E07E0EAFE0FEA7E07EA0E00AC1301EA0F073807FFFC6C13 FE3801FCFC1714809318>I<38FF8FF8A3383800E0A3381C01C0A2137113F9A213D9A2380DDD80 A3138DEA0F8FA23807070015147F9318>119 D<387F8FF000FF13F8007F13F0380E01C0EB0380 A21207EB0700A2EA03871386138EEA01CEA2EA00CCA213DC1378A31370A313F05B1279EA7BC0EA 7F806CC7FC121E151E7F9318>121 D<126012F0B3B012600424769F18>124 D E /Fc 12 119 df97 DI<137E48B4FC38038380EA0F07121E001C1300EA3C0248C7FCA35AA5EA70 021307EA381EEA1FF8EA07E011147C9315>I<1478EB03F814F0EB0070A314E0A4EB01C0A213F1 EA03FD38078F80EA0E07121C123C14001278A3EAF00EA31430EB1C60133CEA707C3878FCC0EA3F CF380F078015207C9F17>I<137C48B4FCEA0783380F0180121E123CEB0300EA780EEA7FFC13E0 00F0C7FCA412701302EA7807EA3C1EEA1FF8EA07E011147C9315>I103 D<136013F0A213E01300A7120FEA1F8012 3113C0EA6380A212C3EA0700A3120EA3EA1C301360A2EA38C01218EA1F80EA0F000C1F7D9E0E> 105 D108 D<381E07C0383F1FE03833B870EA63E0EBC038138000C71370EA0700A3000E 13E0A3EB01C3001C13C6A2EB038C1301003813F8381800F018147D931A>110 D<137C48B4FC38038380380F01C0121E001C13E0123C1278A338F003C0A3EB07801400EA700F13 1EEA3838EA1FF0EA07C013147C9317>I116 D<380F01C0EA1F83003113E0 13C1EA6380A200C313C0EA0700A3380E0180A3EB0300A21306A2EA0F0CEA07F8EA01E013147D93 15>118 D E /Fd 28 121 df<903807F83F017FB512C03A01FC0FE3E03903F01FC7EA07E0D80F C01387ED83C0ED8000A6B612FCA2390FC01F80B2397FF8FFF8A223237FA221>11 D<13181330136013C01201EA0380120713005A121EA2123E123CA2127CA3127812F8AD1278127C A3123CA2123E121EA27E7E13801203EA01C012001360133013180D317BA416>40 D<12C012607E7E121C7E120F7E1380EA03C0A213E01201A213F0A3120013F8AD13F01201A313E0 A2120313C0A2EA078013005A120E5A12185A5A5A0D317DA416>I63 D67 D71 D77 D87 D97 DIII<13FE3807FF80380F87C0381E01E0003E13F0EA7C0014 F812FCA2B5FCA200FCC7FCA3127CA2127E003E13186C1330380FC0703803FFC0C6130015167E95 1A>II<3801FE1F0007B51280380F87E7EA1F03391E01E000003E7FA5001E5BEA1F03 380F87C0EBFF80D819FEC7FC0018C8FC121CA2381FFFE014F86C13FE80123F397C003F8048131F 140FA3007CEB1F00007E5B381F80FC6CB45A000113C019217F951C>II<120E121FEA3F80A3EA1F00 120EC7FCA7EAFF80A2121FB2EAFFF0A20C247FA30F>I108 D<3AFF87F00FE090399FFC3FF83A1FB87E70FC9039E03EC07C9039C03F807EA2018013 00AE3BFFF1FFE3FFC0A22A167E952F>I<38FF87E0EB9FF8381FB8FCEBE07CEBC07EA21380AE39 FFF1FFC0A21A167E951F>I<13FE3807FFC0380F83E0381E00F0003E13F848137CA300FC137EA7 007C137CA26C13F8381F01F0380F83E03807FFC03800FE0017167E951C>I<38FF8FE0EBBFF838 1FF07CEBC03E497E1580A2EC0FC0A8EC1F80A2EC3F00EBC03EEBE0FCEBBFF8EB8FC00180C7FCA8 EAFFF0A21A207E951F>I114 DI<13C0A41201A2120312 07120F121FB5FCA2EA0FC0ABEBC180A51207EBE300EA03FEC65A11207F9F16>I<38FF83FEA238 1F807EAF14FEA2EA0F833907FF7FC0EA01FC1A167E951F>I<39FFF01FE0A2390FC00600A2EBE0 0E0007130CEBF01C0003131813F800015BA26C6C5AA2EB7EC0A2137F6D5AA26DC7FCA2130EA21B 167F951E>I<39FFF07FC0A2390FC01C006C6C5A6D5A6C6C5A00015B3800FD80017FC7FCA27F6D 7E497E80EB67F013E33801C1F8380381FC48C67E000E137E39FF81FFE0A21B167F951E>120 D E /Fe 1 4 df<1207A3EAE738EAFFF8EA7FF0EA1FC0A2EA7FF0EAFFF8EAE738EA0700A30D0E 7E8E12>3 D E /Ff 33 118 df<1318137013E0EA01C0EA0380A2EA0700120EA2121E121C123C A25AA412F85AA97E1278A47EA2121C121E120EA27EEA0380A2EA01C0EA00E0137013180D2D7DA1 14>40 D<12C012707E7E7EA27EEA0380A213C0120113E0A2EA00F0A413F81378A913F813F0A4EA 01E0A213C012031380A2EA0700120EA25A5A5A12C00D2D7DA114>I<14E0B0B712C0A3C700E0C7 FCB022237D9C29>43 D<1238127C12FE12FFA2127F123B1203A21206A2120E120C121812701220 08107C860F>I<137013F0120F12FF12F31203B3A4B51280A2111D7C9C1A>49 DIII<14E0A2497EA3497EA2497EA2 497E130CA2EB187FA201307F143F01707FEB601FA201C07F140F48B57EA2EB800748486C7EA200 06801401000E803AFFE01FFFE0A2231F7E9E28>65 D69 DI73 D78 D82 D<3803FC08380FFF38381E03F8EA3C00481378143812F814187E1400B4FC13F86C B4FC14C06C13E06C13F06C13F8120338001FFC13011300A200C0137CA36C1378A200F813F038FE 01E038E7FFC000811300161F7D9E1D>I<007FB512FCA2397C0FE07C0070141C0060140CA200E0 140E00C01406A400001400B10007B512C0A21F1E7E9D24>I87 D97 D100 DII<3801FC3C3807FFFE380F07DEEA1E03003E13E0A5001E13C0380F 0780EBFF00EA19FC0018C7FCA2121C381FFF8014F06C13F8003F13FC387C007C0070133E00F013 1EA30078133CA2383F01F8380FFFE000011300171E7F931A>II<121C123F5AA37E121CC7FCA6B4FCA2121F B0EAFFE0A20B217EA00E>I108 D<3AFE0FE03F8090393FF0FFC03A1E70F9C3E09039C07F01F0381F807EA2EB007CAC3AFFE3FF8F FEA227147D932C>I<38FE0FC0EB3FE0381E61F0EBC0F8EA1F801300AD38FFE3FFA218147D931D> I<48B4FC000713C0381F83F0383E00F8A248137CA200FC137EA6007C137CA26C13F8A2381F83F0 3807FFC00001130017147F931A>I<38FF1FC0EB7FF0381FE1F8EB80FCEB007EA2143E143FA614 3E147E147CEB80FCEBC1F8EB7FE0EB1F8090C7FCA7EAFFE0A2181D7E931D>I114 DII<38FF07F8A2EA1F00AD1301A2EA0F073807FEFFEA03F818147D931D>I E /Fg 77 124 df<90380FC3E090387FEFF09038E07C783801C0F8D8038013303907007000A7B6 1280A23907007000B0387FE3FFA21D20809F1B>11 DII<90380F80F890387FE7FE9038E06E063901C0 FC0F380380F8380700F00270C7FCA6B7FCA23907007007B03A7FE3FE3FF0A22420809F26>I<90 380FC0FFEB7FE79038E07E0F3801C0FC4848487E38070070A7B7FCA23907007007B03A7FE3FE3F F0A22420809F26>I34 D<127012F812FCA2127C120CA31218A21238123012601240060E7C9F0D >39 D<136013C0EA0180EA03005A12065A121C12181238A212301270A31260A212E0AC1260A212 70A312301238A21218121C120C7E12077EEA0180EA00C013600B2E7DA112>I<12C012607E7E12 1C120C7E12077E1380A2120113C0A31200A213E0AC13C0A21201A313801203A213005A12065A12 1C12185A5A5A0B2E7DA112>I<1306AFB612F0A2D80006C7FCAF1C207D9A23>43 D<127012F812FCA2127C120CA31218A21238123012601240060E7C840D>II<127012F8A3127005057C840D>I<1303A213071306A2130E130CA2131C1318A213381330A2 13701360A213E013C0A212011380A312031300A25A1206A2120E120CA2121C1218A212381230A2 12701260A212E05AA2102D7DA117>IIIII<130EA2131E133EA2136E13EE13CEEA018E120313 0E1206120E120C121812381230126012E0B512F0A238000E00A7EBFFE0A2141E7F9D17>II<137CEA01FEEA07 83380E0380EA0C07121C3838030090C7FC12781270A2EAF3F8EAF7FEEAFC0E487EEB0380A200F0 13C0A51270A214801238EB0700121CEA0E1EEA07FCEA01F0121F7E9D17>I<1260387FFFC0A214 80EA600138C003001306A2C65A5BA25B5BA213E05B1201A3485AA41207A76CC7FC121F7D9D17> II I<127012F8A312701200AA127012F8A3127005147C930D>I<127012F8A312701200AA127012F8 A312781218A41230A21260A21240051D7C930D>I<007FB512E0B612F0C9FCA8B612F06C14E01C 0C7D9023>61 D63 D65 DI<90381FC04090387F F0C03801F8393803C00D38078007380F0003121E003E1301123C127C1400127812F81500A80078 14C0127CA2123C003EEB0180121E6CEB0300EA07803803C00E3801F81C38007FF0EB1FC01A217D 9F21>IIII<90380FC02090387FF8603901F81CE03803E0063807800338 0F0001121E14005A127C1560127812F81500A6EC7FFCA20078EB01E0127CA2123C7EA27E380780 03EA03E03901F80E6039007FFC2090380FE0001E217D9F24>I<39FFF8FFF8A23907800F00AC90 B5FCA2EB800FAD39FFF8FFF8A21D1F7E9E22>II< EAFFFEA2EA0780B11406A4140EA2140C141C143C14FCB5FCA2171F7E9E1C>76 DI<39FF807FF813C0 0007EB07809038E00300A2EA06F0A21378133CA2131EA2130FA2EB078314C31303EB01E3A2EB00 F3A2147BA2143F80A280A2000F7FEAFFF0801D1F7E9E22>III82 D<3807E080EA0FF9EA1C1FEA300FEA7007EA600312E01301A36CC7FCA21278127FEA3FF0EA1FFC 6C7EEA03FF38001F801307EB03C0A2130112C0A400E01380EAF00338F80700EAFE0EEACFFCEA81 F812217D9F19>I<007FB512E0A238780F010070130000601460A200E0147000C01430A4000014 00B23807FFFEA21C1F7E9E21>I<39FFFC7FF8A23907800780EC0300B3A300031302EBC006A200 015B6C6C5AEB7830EB3FE0EB0FC01D207E9E22>I<39FFF007FEA2390F8001F090C712E06C6C13 C0A2EBC00100031480A2EBE00300011400A23800F006A3EB780CA36D5AA36D5AA2EB1F70EB0F60 A214E06D5AA26D5AA31F207F9E22>I<3BFFF07FF83FF0A23B0F0007800F80EE0300A23A07800F C006A3913819E00ED803C0140CA214393A01E030F018A33A00F0607830A3ECE07C903978C03C60 A390393D801EC0A390383F000F6D5CA3010E6DC7FCA32C207F9E2F>I<397FF83FF8A23907C00F 800003EB06003801E00EEBF00C00005BEB7838EB7C30EB3C70EB3E60EB1EC0130F5C1307808013 0DEB1DF0EB18F8EB3878EB307CEB603CEBE01EEBC01F48487E0003EB0780010013C0EA0F8039FF E01FFEA21F1F7F9E22>I92 D97 D<120E12FEA2120EA9133FEB FF80380FC3C0EB00E0000E13F014701478A7147014F0120FEB01E0EBC3C0380CFF80EB3E001520 7F9F19>IIII<133C13FEEA01CFEA038F1306EA0700A7EAFFF0A2EA0700B0EA7FF0A21020 809F0E>II<120E12FEA2120EA9133E13FF380FC380EB01C0 A2120EAD38FFE7FCA216207F9F19>I<121C121E123E121E121CC7FCA6120E127EA2120EAFEAFF C0A20A1F809E0C>I<13E0EA01F0A3EA00E01300A61370EA07F0A212001370B3A21260EAF0E0EA F1C0EA7F80EA3E000C28829E0E>I<120E12FEA2120EA9EB1FF0A2EB0F80EB0E00130C5B5B1370 13F0EA0FF81338EA0E1C131E130E7F1480130314C038FFCFF8A215207F9F18>I<120E12FEA212 0EB3A9EAFFE0A20B20809F0C>I<390E3F03F039FEFF8FF839FFC1DC1C390F80F80EEB00F0000E 13E0AD3AFFE7FE7FE0A223147F9326>IIII<3803E180EA0FF9EA1E1FEA3C0712781303127012F0A6127012781307EA3C0FEA 1E1FEA0FF3EA03E3EA0003A7EB3FF8A2151D7E9318>III<1206A4120EA2121E123E EAFFF8A2EA0E00AA1318A5EA073013E0EA03C00D1C7F9B12>I<380E01C0EAFE1FA2EA0E01AC13 03A2EA070FEBFDFCEA01F116147F9319>I<38FF87F8A2381E01E0000E13C01480A238070300A3 EA0386A2138EEA01CCA213FC6C5AA21370A315147F9318>I<39FF9FF3FCA2391C0780F01560EC C0E0D80E0F13C0130C14E00007EBE180EB186114713903987300EBB033A2143F3801F03EEBE01E A20000131CEBC00C1E147F9321>I<387FC7FCA2380703E0148038038300EA01C7EA00EE13EC13 781338133C137C13EEEA01C7138738030380380701C0000F13E038FF87FEA21714809318>I<38 FF87F8A2381E01E0000E13C01480A238070300A3EA0386A2138EEA01CCA213FC6C5AA21370A313 60A35B12F0EAF18012F3007FC7FC123C151D7F9318>III E /Fh 29 122 df<48B4FC000F13E0383E03F8007813FCEA7E01 00FF13FEA5387E03FC1200EB07F0EB0FE01480EB1F00131E5BA25BA21370A690C7FCA61370EA01 FC487EA56C5AEA0070172A7CA920>63 D 70 D<91387FE003903903FFFC0F011FEBFF1F90397FF00FFF9038FF8001D803FEC7FC48488048 48804980485A003F815B007F81A3484891C7FCA90203B512F8A2EA7FC0DA00011300A2123F7F12 1F6C7E7F6C7E6C6C5B3800FF8090387FF00F011FB5123F0103EBFC0F9039007FE0032D297CA836 >I73 D77 D<90387F80603903FFF0E0000F13FF38 1F807F383F001F003E1307007E1303127C00FC1301A214007E7E6D130013F8EBFF806C13F814FE 6C7F6C14C07E6C14E0000114F0EA003F010113F8EB001F1407A200E013031401A37E15F06C1303 6C14E0B413079038E01FC090B5120000E05B38C01FF01D297CA826>83 D87 D<48B47E000F13F0381F 81FC486C7E147FA2EC3F80A2EA0F00C7FCA2EB0FFF90B5FC3807FC3FEA1FE0EA3F80127F130012 FEA3147F7E6CEBFFC0393F83DFFC380FFF0F3801FC031E1B7E9A21>97 D99 DIII<9038FF81F00003EB E7FC390FC1FE7C391F80FCFC003FEBFE7C9038007E3848EB7F00A66C137EEB80FE001F5B380FC1 F8381FFFE0001813800038C8FC123CA2123E383FFFF814FF6C14C06C14E06C14F0121F397E0007 F8007C13015A1400A36C1301007EEB03F06CEB07E0390FC01F803903FFFE0038007FF01E287E9A 22>II<1207EA1FC013E0123FA3121F13C0EA0700C7FCA7EAFFE0A3 120FB3A3EAFFFEA30F2B7DAA14>I107 DI<3BFFC07F800FF0 903AC1FFE03FFC903AC783F0F07E3B0FCE03F9C07F903ADC01FB803F01F8D9FF00138001F05BA3 01E05BAF3CFFFE1FFFC3FFF8A3351B7D9A3A>I<38FFC07F9038C1FFC09038C787E0390FCE07F0 9038DC03F813F813F0A313E0AF3AFFFE3FFF80A3211B7D9A26>II<38FFE1FE9038E7FF809038FE07E039 0FF803F8496C7E01E07F140081A2ED7F80A9EDFF00A25DEBF0014A5A01F85B9038FE0FE09038EF FF80D9E1FCC7FC01E0C8FCA9EAFFFEA321277E9A26>I<38FFC3F0EBCFFCEBDC7E380FD8FF13F8 5BA3EBE03C1400AFB5FCA3181B7E9A1C>114 D<3803FE30380FFFF0EA3E03EA7800127000F013 70A27E6C1300EAFFE013FE387FFFC06C13E06C13F0000713F8C613FC1303130000E0137C143C7E A26C13787E38FF01F038F7FFC000C11300161B7E9A1B>I<1370A413F0A312011203A21207381F FFF0B5FCA23807F000AD1438A73803F870000113F03800FFE0EB1F8015267FA51B>I<39FFE03F F8A3000F1303B11407A2140F0007131F3A03F03BFF803801FFF338003FC3211B7D9A26>I<3AFF FE03FF80A33A07F0007000A26D13F000035CEBFC0100015CA26C6C485AA2D97F07C7FCA2148FEB 3F8E14DEEB1FDCA2EB0FF8A36D5AA26D5AA26D5A211B7F9A24>I<3BFFFE7FFC0FFEA33B0FE007 E000E03B07F003F001C0A29039F807F80300031680A23B01FC0EFC0700A2D9FE1E5B000090381C 7E0EA29039FF383F1E017F141C0278133C90393FF01FB8A216F86D486C5AA26D486C5AA36D486C 5AA22F1B7F9A32>I<39FFFC0FFFA33907F003C06C6C485AEA01FC6C6C48C7FCEBFF1E6D5AEB3F F86D5A130FA2130780497E497E131EEB3C7F496C7E496C7ED801E07FEBC00F00036D7E3AFFF01F FF80A3211B7F9A24>I<3AFFFE03FF80A33A07F0007000A26D13F000035CEBFC0100015CA26C6C 485AA2D97F07C7FCA2148FEB3F8E14DEEB1FDCA2EB0FF8A36D5AA26D5AA26D5AA2495AA2EA3807 007C90C8FCEAFE0F130E131E5BEA7C78EA3FE0EA0FC021277F9A24>I E /Fi 18 119 df<127012F812FCA2127C120CA41218A21230A212601240060F7C840E>44 D49 DI56 DI77 D<39FF800FFF13C00007EB01F89038E000607F12061378A27F133E13 1E7FA2EB078014C01303EB01E0A2EB00F01478A2143CA2141E140FA2EC07E0A214031401A2381F 8000EAFFF0156020227EA125>I<3803F020380FFC60381C0EE0EA3803EA7001A2EAE000A21460 A36C1300A21278127FEA3FF0EA1FFE6C7E0003138038003FC0EB07E01301EB00F0A2147012C0A4 6C136014E06C13C0EAF80138EF038038C7FF00EA81FC14247DA21B>83 D97 D<120E12FEA2121E120EAAEB1F80EB7FE0380FC0F0EB0078000E1338143C 141C141EA7141C143C000F1338EB8070EBC1F0380C7FC0EB1F0017237FA21B>II101 D<121C121E123E121E121CC7FC A8120E12FEA2121E120EAFEAFFC0A20A227FA10E>105 D<390E1FC07F3AFE7FE1FF809039C0F3 03C03A1F807E01E0390F003C00000E1338AE3AFFE3FF8FFEA227157F942A>109 D<380E1F8038FE7FC038FFC1E0381F80F0380F0070120EAE38FFE7FFA218157F941B>II114 D<38FFC3FEA2381E00F8000E1360A26C13C0A338038180 A213C300011300A2EA00E6A3137CA31338A217157F941A>118 D E /Fj 17 124 df73 D77 D79 DI97 D 99 D101 DI<15F090387F03F83901FFCF1C3803C1FC390780F818390F007800 48137C001E133C003E133EA7001E133C001F137C6C13786C6C5A380FC1E0380DFFC0D81C7FC7FC 0018C8FCA2121CA2121E380FFFF814FF6C14804814C0391E0007E00038EB01F048EB0070157848 1438A500701470007814F06CEB01E06CEB03C03907C01F003801FFFC38003FE01E2F7E9F21>I< 1207EA0F80121FA2120FEA0700C7FCABEA078012FFA2120F1207B3A6EA0FC0EAFFF8A20D307EAF 12>105 D108 D<380781FE39FF87FF8090388E07C0390F9803E03807B0019038E000F05BA35BB3486C487E3AFF FC1FFF80A2211F7E9E25>110 D<380783E038FF8FF8EB9C7CEA0FB0EA07F0EBE038EBC000A35B B3487EEAFFFEA2161F7E9E19>114 D<3801FC10380FFF30381E03F0EA38004813705A1430A37E 6C1300127EEA3FF06CB4FC6C1380000313E038003FF0EB03F8EB007800C0133CA2141C7EA27E14 186C13386C137038EF01E038C3FFC03880FE00161F7E9E1A>I<13C0A51201A31203A21207120F 121FB512E0A23803C000B01430A83801E060A23800F0C0EB7F80EB1F00142C7FAB19>II123 D E end %%EndProlog %%BeginSetup %%Feature: *Resolution 300 TeXDict begin %%EndSetup %%Page: 1 1 bop 457 227 a Fj(Message)21 b(P)n(assing)g(In)n(terface)i({)f(Outline)869 354 y Fi(Marc)16 b(Snir)772 456 y(No)o(v)o(em)o(b)q(er)d(28,)k(1992)75 643 y Fh(In)n(tro)r(duction)75 745 y Fg(The)f(purp)q(ose)h(of)f(this)g(do)q (cumen)o(t)h(is)f(to)g(pro)o(vide)g(a)g(framew)o(ork)f(that)g(w)o(e)h(can)g (use)h(to)e(discuss)i(issues)75 801 y(in)i(the)e(de\014nition)j(of)d(MPI,)h (to)f(list)i(questions)f(w)o(e)f(need)i(to)e(answ)o(er,)h(and)g(prop)q(ose)f (some)h(answ)o(ers.)75 857 y(I)g(plan)g(to)e(use)i(this)g(do)q(cumen)o(t)f (as)g(a)g(framew)o(ork)f(for)h(the)g(discussions)i(in)f(the)f(p)q(oin)o (t-to-p)q(oin)o(t)h(sub-)75 914 y(committee;)d(I)g(broadcast)g(it)g(to)g(the) g(en)o(tire)g(committee,)g(b)q(ecause)h(man)o(y)f(issues)h(are)f(relev)m(an)o (t)g(to)g(the)75 970 y(other)g(sub)q(committees.)75 1113 y Fh(Goals)131 1214 y Fg(1.)22 b(Design)17 b(an)h(application)h(programming)d (in)o(terface)i(\(not)e(necessarily)j(a)e(target)f(for)h(compilers)189 1271 y(or)d(a)h(system)g(implemen)o(tation)h(library\))131 1363 y(2.)22 b(Allo)o(w)15 b(e\016cien)o(t)h(comm)o(unication:)21 b(Av)o(oid)16 b(memory)e(to)h(memory)g(cop)o(ying)g(and)h(allo)o(w)f(o)o(v)o (erlap)189 1420 y(of)e(computation)g(and)h(comm)o(unication)g(and)g(o\017oad) f(to)g(comm)o(unication)h(copro)q(cessor,)g(where)189 1476 y(a)o(v)m(ailable)131 1569 y(3.)22 b(Allo)o(w)15 b(\(but)g(no)h(mandate\))e (extensions)i(for)f(use)g(in)h(heterogeneous)f(en)o(vironmen)o(t.)131 1661 y(4.)22 b(Allo)o(w)15 b(con)o(v)o(enien)o(t)h(C,)f(C++,)g(F77)f(and)i (F90)e(bindings)j(for)e(in)o(terface.)131 1754 y(5.)22 b(Pro)o(vide)15 b(a)g(reliable)i(comm)o(unication)f(in)o(terface:)21 b(User)15 b(need)h(not)f(cop)q(e)h(with)f(comm)o(unication)189 1810 y(failures.)21 b(Suc)o(h)15 b(failures)i(are)d(dealt)i(b)o(y)f(the)h(underlying)h(comm)o (unication)f(subsystem.)131 1902 y(6.)22 b(F)l(o)q(cus)15 b(on)g(a)g(prop)q (osal)h(that)e(can)h(b)q(e)h(agreed)f(up)q(on)h(in)g(6)f(mon)o(ths.)131 1995 y(7.)22 b(De\014ne)10 b(an)g(in)o(terface)h(that)e(is)i(not)f(to)q(o)f (di\013eren)o(t)i(from)e(curren)o(t)h(practice)h(\(PVM/Express/P)o(armacs\)) 131 2087 y(8.)22 b(De\014ne)13 b(an)g(in)o(terface)f(that)g(can)h(b)q(e)h (quic)o(kly)g(implemen)o(ted)g(on)f(man)o(y)f(v)o(endors)g(platforms,)h(with) 189 2144 y(no)i(signi\014can)o(t)h(c)o(hanges)f(in)h(the)f(underlying)j(comm) o(unication)d(and)h(system)f(soft)o(w)o(are.)146 2245 y(Do)f(w)o(e)h(agree)g (on)g(this?)75 2388 y Fh(F)-6 b(ramew)n(ork)75 2489 y Fg(I)11 b(prop)q(ose)g(to)g(consider)g(comm)o(unication)h(op)q(erations)f(as)g (consisting)g(of)g(the)g(follo)o(wing)g(sub)q(op)q(erations:)75 2591 y Ff(INIT\(op)q(eration,)19 b(params,)d(handle\))24 b Fg(Pro)q(cess)15 b(pro)o(vides)f(all)i(relev)m(an)o(t)e(parameters)g(for)g (its)g(par-)189 2647 y(ticipation)19 b(in)f(the)g(comm)o(unication)g(op)q (eration)g(\(data)f(bu\013er,)h(t)o(yp)q(e,)g(participan)o(ts,)g(etc.\).)27 b(A)189 2704 y(handle)16 b(is)g(created)f(that)g(iden)o(ti\014es)h(the)g(op)q (eration.)964 2828 y(1)p eop %%Page: 2 2 bop 75 45 a Ff(ST)l(AR)l(T\(handle\))24 b Fg(The)16 b(comm)o(unication)g(op)q (eration)f(is)h(started)75 139 y Ff(A)-6 b(W)g(AIT\(handle\))24 b Fg(The)15 b(comm)o(unication)h(op)q(eration)g(is)g(completed.)75 232 y Ff(FREE\(handle\))25 b Fg(The)16 b(handle,)g(and)f(asso)q(ciated)g (resources)h(are)f(freed.)75 337 y(Correct)g(in)o(v)o(o)q(cation)h(of)g (these)g(sub)q(op)q(erations)g(is)h(a)e(sequence)i(of)e(the)h(form)f(1\(23\)) 1539 321 y Fe(\003)1558 337 y Fg(4.)21 b(I.e.,)15 b(a)h(handle)75 394 y(need)g(b)q(e)g(created)g(b)q(efore)g(comm)o(unication)g(o)q(ccurs;)g (it)f(can)h(b)q(e)g(reused)g(only)g(after)f(the)g(previous)i(use)75 450 y(has)12 b(completed;)h(and)f(it)g(need)h(to)e(b)q(e)h(freed)g(ev)o(en)o (tually)h(\(of)e(course,)h(one)g(can)g(assume)f(that)g(all)i(handles)75 507 y(are)i(freed)g(at)g(program)f(termination,)h(b)o(y)g(default\).)146 563 y(The)f(c)o(hoice)i(of)e(these)h(sub)q(op)q(erations)g(is)h(somewhat)d (arbitrary)l(.)20 b(The)15 b(justi\014cation)g(is)g(that)f(com-)75 619 y(m)o(unication)k(ma)o(y)f(b)q(e)h(a)g(length)o(y)g(pro)q(cess)f(one)h (desires)h(to)d(o)o(v)o(erlap)i(with)g(computation,)f(hence)i(the)75 676 y(separation)g(of)g(2)h(and)f(3;)i(also)e(comm)o(unication)i(setup)e(ma)o (y)g(in)o(v)o(olv)o(e)h(a)f(signi\014can)o(t)i(o)o(v)o(erhead)e(one)75 732 y(desires)i(to)f(amortize)h(o)o(v)o(er)e(man)o(y)h(successiv)o(e)i(comm)o (unications)f(with)g(the)g(same)f(c)o(haracteristics,)75 789 y(hence)c(the)g(separation)f(of)f(1)h(and)h(4)f(from)f(2)h(and)g(3.)75 932 y Fh(Issues)75 1035 y Fd(Whic)n(h)20 b(op)r(erations?)75 1121 y Fg(SEND)15 b(and)g(RECEIVE)g(for)g(p)q(oin)o(t)g(to)f(p)q(oin)o(t)i (comm)o(unication.)k(Longer)15 b(list)h(for)e(collectiv)o(e)j(comm)o(u-)75 1177 y(nication)g(\(note:)k(SEND)16 b(and)g(RECEIVE)g(are)g(particular)g (case)g(of)g(broadcast,)f(for)g(group)g(of)h(size)h(2;)75 1234 y(this)i(observ)m(ation)g(can)g(b)q(e)g(used)h(to)e(c)o(hec)o(k)h(if)g (de\014nition)i(of)d(collectiv)o(e)i(comm)o(unication)g(seman)o(tics)75 1290 y(are)15 b(consisten)o(t)g(with)h(de\014nition)h(of)e(p)q(oin)o(t-to-p)q (oin)o(t)h(comm)o(unication\).)146 1347 y(One)g(can)f(argue)g(for)f(a)h(SEND) p 684 1347 14 2 v 17 w(RECV)g(\(or)f(EX)o(CHANGE\))h(op)q(eration.)146 1403 y(I)g(prop)q(ose)g(to)g(lea)o(v)o(e)g(out,)f(for)g(the)i(time)f(b)q (eing,)h(things)f(lik)o(e)i(remote)d(pro)q(cedure)i(calls,)g(monitors,)75 1460 y(atomic)h(op)q(erations,)i(semaphores,)e(activ)o(e)h(messages,)g (message)f(handlers,)i(etc.)27 b(This,)19 b(in)f(order)g(to)75 1516 y(fo)q(cus)f(initially)j(on)c(a)h(smaller)g(set)g(of)f(issues,)i(with)f (less)g(dep)q(endencie)q(s)i(on)e(the)g(underlying)i(pro)q(cess)75 1572 y(mo)q(del.)146 1629 y(Opinions?)75 1750 y Fd(What)g(calls)h(are)e(made) h(a)n(v)m(ailable)h(to)f(the)f(user?)75 1836 y Fg(Options:)75 1941 y Ff(\(1+2+3+4\))23 b Fg(blo)q(c)o(king)16 b(send/receiv)o(e)75 2035 y Ff(\(1+2\))i(\(3+4\))23 b Fg(non)o(blo)q(c)o(king)17 b(send/receiv)o(e,)f(follo)o(w)o(ed)f(b)o(y)h(w)o(ait)75 2128 y Ff(\(1\))i(\(2+3\))g(\(4\))24 b Fg(c)o(hannel)16 b(creation;)f(blo)q(c)o (king)i(c)o(hannel)f(send/receiv)o(e;)g(c)o(hannel)h(cancelation)75 2222 y Ff(\(1\))h(\(2\))g(\(3\))h(\(4\))k Fg(c)o(hannel)17 b(creation;)e(non)o(blo)q(c)o(king)h(send/receiv)o(e;)g(w)o(ait;)f(c)o (hannel)h(destruction)146 2327 y(An)o(y)f(more)g(needed?)22 b(An)o(y)15 b(less?)75 2448 y Fd(When)j(is)i(a)f(comm)n(unication)h(op)r (eration)e(completed?)75 2534 y Fg(The)12 b(main)f(distinction)j(is)e(b)q(et) o(w)o(een)f Fc(lo)n(c)n(al)g Fg(termination)g(and)h Fc(glob)n(al)f Fg(termination.)19 b(A)11 b(send)h(terminates)75 2591 y(lo)q(cally)23 b(when)f(data)f(has)h(b)q(een)h(copied)g(out)e(of)g(the)h(sender)g(bu\013er)g (\(formally)l(,)h(when)f(the)f(sender)75 2647 y(cannot)e(a\013ect)g(an)o (ymore)g(the)h(outcome)g(of)f(the)h(send)g(op)q(eration\).)33 b(It)20 b(terminates)g(globally)g(when)75 2704 y(data)d(has)h(b)q(een)h (receiv)o(ed)g(b)o(y)f(the)f(receiv)o(er)i(\(formally)l(,)f(when)h(receiv)o (er)f(has)g(terminated)g(executing)964 2828 y(2)p eop %%Page: 3 3 bop 75 45 a Fg(an)o(y)15 b(op)q(eration)h(that)f(logically)j(precedes)f(the)f (execution)g(of)g(the)g(receiv)o(e\).)22 b(The)16 b(distinction)h(has)f(no)75 102 y(meaning)g(for)e(a)h(receiv)o(e)h({)f(since)i(lo)q(cal)f(termination)g (of)e(a)h(receiv)o(e)h(implies)i(global)d(termination.)146 158 y(Choices:)131 252 y(1.)22 b(T)l(ermination)16 b(is)f(alw)o(a)o(ys)g(lo)q (cal.)131 346 y(2.)22 b(T)l(ermination)e(is)g(either)g(lo)q(cal)h(or)e (global,)i(at)e(user)g(c)o(hoice)i(\(e.g.,)e(user)g(can)h(c)o(ho)q(ose)g(b)q (et)o(w)o(een)189 402 y(sync)o(hronous)15 b(and)g(async)o(hronous)g(comm)o (unication\))131 496 y(3.)22 b(T)l(ermination)16 b(is)f(alw)o(a)o(ys)g (global.)146 590 y(The)21 b(curren)o(t)f(MPI)h(prop)q(osal)g(c)o(ho)q(oses)g (2.)37 b(If)21 b(w)o(e)g(stic)o(k)g(to)f(this)h(c)o(hoice)h(then)f(I)g (suggest)g(that)75 646 y(termination)h(mo)q(de)g(\(global)g(or)f(lo)q(cal\))h (is)g(part)f(of)g(the)h(op)q(eration)f(parameters)g(that)g(are)g(set)g(up)75 703 y(in)g(sub)q(op)q(eration)g(1.)34 b(F)l(urthermore,)21 b(I)f(w)o(ould)h(suggest)e(to)h(o\013er)f(the)h(same)g(option)h(for)e (collectiv)o(e)75 759 y(comm)o(unications.)146 816 y(Another)13 b(issue)h(is)g(whether)g(termination)g(mo)q(de)f(need)h(b)q(e)h(consisten)o (t)e(for)g(all)h(participan)o(ts.)20 b(Cur-)75 872 y(ren)o(t)c(MPI)g(prop)q (osal)g(requires)h(that)e(a)g(sync)o(hronous)h(send)h(b)q(e)g(matc)o(hed)e(b) o(y)h(a)g(sync)o(hronous)g(receiv)o(e)75 928 y(\(either)i(all)h(participan)o (ts)f(use)g(lo)q(cal)h(completion)g(or)e(all)i(use)f(global)g(completion\).) 29 b(There)18 b(seem)g(to)75 985 y(b)q(e)d(no)g(comp)q(elling)i(logical)f (reason)f(for)f(this)h(restriction,)g(and)g(I)g(am)g(not)f(sure)h(it)g(has)g (signi\014can)o(t)g(im-)75 1041 y(plemen)o(tation)i(adv)m(an)o(tages.)k(The) 16 b(alternativ)o(e)g(is)g(to)f(allo)o(w)h(eac)o(h)g(participan)o(t)g(in)h(a) e(comm)o(unication)75 1098 y(op)q(eration)h(to)g(c)o(ho)q(ose)g(either)h(lo)q (cal)h(or)e(global)h(termination)f(\(this)h(b)q(ecomes)g(more)e(signi\014can) o(t)j(for)d(a)75 1154 y(collectiv)o(e)i(op)q(eration,)e(where)g(the)h(c)o (hoice)g(is)g(meaningful)g(at)f(more)f(than)i(one)f(pro)q(cess\).)146 1211 y(I)i(assume)h(lo)q(cal)g(termination)g(for)f(sub)q(op)q(eration)h(1)f ({)g(no)h(sync)o(hronization)g(with)g(another)f(pro-)75 1267 y(cessor)e(is)h(required)g(for)f(this)g(phase.)146 1324 y(Opinions?)146 1380 y(Should)h(w)o(e)f(c)o(ho)q(ose)g(1)g(or)g(3,)f(rather)h(than)g(2?)146 1437 y(Should)g(w)o(e)f(use)g(another)g(mec)o(hanism)g(to)g(c)o(ho)q(ose)g (lo)q(cal)h(and)f(global)h(termination)f(\(e.g.,)f(a)g(global)75 1493 y(\015ag\)?)146 1549 y(Should)g(w)o(e)e(require)h(the)g(c)o(hoice)h(of)e (\\sync)o(hronous")g(to)g(b)q(e)h(done)g(consisten)o(tly)h(b)q(e)f(all)h (participan)o(ts)75 1606 y(in)j(a)f(comm)o(unication?)75 1728 y Fd(When)j(do)r(es)g(a)h(call)h(return?)75 1813 y Fg(The)14 b(ob)o(vious)f(c)o(hoice)i(is:)k(\\A)13 b(call)i(return)e(when)h(the)f (corresp)q(onding)i(sub)q(op)q(eration\(s\))e(terminated".)75 1870 y(An)21 b(alternativ)o(e)f(is)h(to)f(allo)o(w)h(for)f(unsuccessful)i (return)f(as)f(w)o(ell.)37 b(E.g.,)20 b(send)h(can)g(return)f(unsuc-)75 1926 y(cessfully)l(,)d(if)f(the)f(system)g(cannot)g(accept)g(the)h(message,)e (for)h(whatev)o(er)g(reasons;)f(or)h(receiv)o(e)h(returns)75 1983 y(unsuccessfully)f(if)e(the)g(receiv)o(e)h(cannot)e(execute)i(within)g (a)e(set)h(time;)g(and)g(so)g(on.)19 b(This)13 b(is)g(the)g(curren)o(t)75 2039 y(MPI)i(de\014nition.)146 2096 y(My)j(p)q(ersonal)i(c)o(hoice)g(is)g (against)e(unsuccessful)j(return.)32 b(This)20 b(is)f(more)g(adequate)g(for)f (system)75 2152 y(programming,)f(rather)h(than)f(application)j(programming;)e (it)g(in)o(tro)q(duces)h(seman)o(tic)f(dep)q(endencies)75 2209 y(on)h(implemen)o(tation;)i(it)e(forces)f(the)h(user)f(to)g(c)o(hec)o(k)h (for)f(success)i(at)e(eac)o(h)g(op)q(eration,)i(rather)e(then)75 2265 y(deal)j(with)g(failure)g(of)f(comm)o(unication)h(using)g(an)f (exception)i(handling)g(mec)o(hanism.)36 b(Of)20 b(course,)75 2322 y(implemen)o(ters)f(are)f(w)o(elcome)g(to)f(pro)o(vide)i(a)e(reasonable) i(exception)f(handling)i(mec)o(hanism)f(for)e(the)75 2378 y(MPI)e(library)h ({)f(I)h(suggest)e(to)h(lea)o(v)o(e)g(op)q(en)h(the)f(de\014nition)i(of)e (suc)o(h)h(mec)o(hanism.)146 2434 y(Opinions?)75 2556 y Fd(Correctness)i (criteria)75 2642 y Fg(I)e(use)f(the)g(follo)o(wing)h(terminology:)964 2828 y(3)p eop %%Page: 4 4 bop 75 45 a Ff(safe)17 b(program)22 b Fg(A)c(program)f(that)h(should)h (execute)f(successfully)i(in)f(an)o(y)f(implemen)o(tation,)i(inde-)189 102 y(p)q(enden)o(tly)c(of)f(the)h(amoun)o(t)e(of)h(a)o(v)m(ailable)i(system) d(resources.)75 195 y Ff(unsafe)j(program)23 b Fg(A)d(program)f(that)g(will)j (succeed)f(or)f(fail,)i(dep)q(ending)g(on)e(implemen)o(tation)h(or)189 252 y(a)o(v)m(ailable)c(system)d(resources.)75 346 y Ff(erroneous)i(program) 22 b Fg(A)16 b(program)e(that)g(will)j(alw)o(a)o(ys)e(fail.)146 452 y(Examples)g(\(in)o(v)o(olving)h(t)o(w)o(o)e(pro)q(cessors)h(with)h(ids)g (1)f(and)g(2,)f(and)i(using)g(MPI)f(lik)o(e)h(syn)o(tax\))146 508 y(The)f(follo)o(wing)h(program)e(is)i(safe,)e(and)i(should)g(alw)o(a)o (ys)e(succeed.)75 659 y Fb(IF)24 b(\(GETID\(\))e(==)i(1\))g(THEN)147 715 y(CALL)f(CSEND\(mode=blocking,)e(buf=sendbuf,)h(dest=2,)h(type=0,)g (len=1000\))147 772 y(CALL)g(CRECV\(mode=blocking,)e(buf=recdbuf,)h (source=2,)h(type=0,)g(len=1000\))75 884 y(ELSE)g(!)h(\(GETID\(\))f(==)g(2\)) 147 941 y(CALL)g(CRECV\(mode=blocking,)e(buf=recbuf,)h(source=1,)h(type=0,)g (len=1000\))147 997 y(CALL)g(CSEND\(mode=blocking,)e(buf=sendbuf,)h(dest=1,)h (type=0,)g(len=1000\))146 1148 y Fg(The)15 b(follo)o(wing)h(program)e(is)i (erroneous,)e(and)i(should)g(alw)o(a)o(ys)e(fail.)75 1310 y Fb(IF)24 b(\(GETID\(\))e(==)i(1\))g(THEN)147 1367 y(CALL)f (CSEND\(mode=synchronous,)e(buf=sendbuf,)h(dest=2,)h(type=0,)g(len=1000\))147 1423 y(CALL)g(CRECV\(mode=synchronous,)e(buf=recdbuf,)h(source=2,)g(type=0,)h (len=1000\))75 1536 y(ELSE)47 b(!)24 b(\(GETID\(\))f(==)g(2\))147 1593 y(CALL)g(CSEND\(mode=synchronous,)e(buf=sendbuf,)h(dest=1,)h(type=0,)g (len=1000\))147 1649 y(CALL)g(CRECV\(mode=synchronous,)e(buf=recbuf,)h (source=1,)h(type=0,)g(len=1000\))146 1755 y Fg(The)12 b(follo)o(wing)h (program)d(is)j(unsafe,)f(and)h(ma)o(y)e(succeed)i(or)f(fail,)h(dep)q(ending) h(on)e(implemen)o(tation.)75 1918 y Fb(IF)24 b(\(GETID\(\))e(==)i(1\))g(THEN) 147 1975 y(CALL)f(CSEND\(mode=blocking,)e(buf=sendbuf,)h(dest=2,)h(type=0,)g (len=1000000\))147 2031 y(CALL)g(CRECV\(mode=blocking,)e(buf=recdbuf,)h (source=2,)h(type=0,)g(len=1000000\))75 2144 y(ELSE)g(!)h(\(GETID\(\))f(==)g (2\))147 2200 y(CALL)g(CSEND\(mode=blocking,)e(buf=sendbuf,)h(dest=1,)h (type=0,)g(len=1000000\))147 2257 y(CALL)g(CRECV\(mode=blocking,)e (buf=recbuf,)h(source=1,)h(type=0,)g(len=1000000\))146 2363 y Fg(This)14 b(program)f(can)h(succeed)h(only)g(if)f(the)g(system)g(has)g (su\016cien)o(t)g(bu\013er)g(space)g(to)g(bu\013er)g(1)f(MgB)75 2420 y(of)i(data.)146 2476 y(The)j(strict)g(de\014nition)i(of)e(\\safe")f (needs)i(to)f(b)q(e)g(somewhat)g(temp)q(ered)h(in)g(practice.)29 b(A)18 b(correct)75 2532 y(sequen)o(tial)13 b(program)d(ma)o(y)h(fail)i(b)q (ecause)f(of)g(time)g(limits)h(or)e(other)g(resource)h(restrictions.)19 b(Lik)o(ewise,)13 b(a)75 2589 y(safe)h(parallel)i(program)d(ma)o(y)l(,)g (exceptionally)l(,)k(exceed)e(system)f(resources.)19 b(Still,)d(common)e (sense)h(can)75 2645 y(di\013eren)o(tiate)i(b)q(et)o(w)o(een)g(resources)f (that)g(are)h(\\practically)g(un)o(b)q(ounded",)h(i.e.)25 b(with)17 b(limits)g(that)f(are)75 2702 y(unlik)o(ely)k(to)d(b)q(e)h(normally)g(encoun) o(tered)g(b)o(y)g(correct)f(programs,)g(and)g(resources)h(that)f (\\practically)964 2828 y(4)p eop %%Page: 5 5 bop 75 45 a Fg(b)q(ounded",)23 b(i.e.)37 b(with)21 b(limits)h(one)f(encoun)o (ters)g(frequen)o(tly)g(with)g(correct)g(programs,)f(and)h(whic)o(h)75 102 y(often)13 b(impinges)i(on)f(program)e(p)q(ortabilit)o(y)l(.)20 b(A)14 b(formal)f(de\014nition)i(will)h(assume)d(that)g(some)g(resources)75 158 y(are)19 b(unlimited,)k(and)c(implemen)o(tations)i(will)g(striv)o(e)f(to) f(pro)o(vide)h(a)f(reasonable)h(appro)o(ximation)f(of)75 214 y(this)f(assumption.)26 b(F)l(urthermore,)17 b(the)g(user)h(should)g(b)q(e)g (able)g(to)f(query)g(and)g(p)q(ossibly)i(con)o(trol)e(an)o(y)75 271 y(implemen)o(tation)f(sp)q(eci\014c)i(resource)d(limitation)h(that)f (a\013ects)f(program)g(b)q(eha)o(vior.)146 327 y(One)k(need)g(to)f(allo)o(w)h (\(indeed,)h(one)f(cannot)f(disallo)o(w\))h(v)o(endors)g(to)e(supp)q(ort)i (some)f(unsafe)h(pro-)75 384 y(grams)c(as)h(w)o(ell.)21 b(The)15 b(requiremen)o(ts)h(there)f(should)h(b)q(e)g(that)143 478 y Fa(\017)23 b Fg(The)15 b(seman)o(tics)g(of)g(successful)i(program)d (execution)i(b)q(e)g(w)o(ell-de\014ned)143 571 y Fa(\017)23 b Fg(The)16 b(system)f(pro)o(vide)i(clear)f(information)h(and)f(p)q(ossibly)h (con)o(trol)f(on)g(those)g(parameters)f(that)189 628 y(determine)h(the)f(b)q (eha)o(vior)h(of)f(unsafe)g(programs)f(\(e.g.,)g(bu\013er)h(sizes\).)146 722 y(A)g(handle)j(is)e Fc(active)g Fg(\(at)f(a)g(pro)q(cess\))h(from)f(the)h (start)f(of)g(sub)q(op)q(eration)i(1)e(to)g(the)h(completion)h(of)75 778 y(sub)q(op)q(eration)f(4)g(for)f(this)h(handle.)22 b(A)15 b(comm)o(unication)i(is)f Fc(active)f Fg(\(at)g(a)g(pro)q(cess\))h(from)e (the)i(start)e(of)75 835 y(sub)q(op)q(eration)i(2)f(to)g(the)g(completion)h (of)f(sub)q(op)q(eration)h(3.)k(I)15 b(suggest)g(the)g(follo)o(wing)h(design) g(p)q(oin)o(t:)131 941 y(1.)22 b(The)13 b(n)o(um)o(b)q(er)g(of)g(activ)o(e)g (comm)o(unication)g(handles)i(p)q(er)e(pro)q(cess)g(is)h(\\practically)g(un)o (b)q(ounded".)131 1035 y(2.)22 b(The)d(n)o(um)o(b)q(er)g(of)f(initiated)j (comm)o(unications)e(p)q(er)h(pro)q(cessor)f(is)g(\\practically)h(un)o(b)q (ounded".)189 1091 y(\(Note)12 b(that)h(there)g(ma)o(y)g(b)q(e)g(only)h(one)g (activ)o(e)f(comm)o(unication)h(op)q(eration)f(p)q(er)h(activ)o(e)f (handle\).)131 1185 y(3.)22 b(The)16 b(amoun)o(t)g(of)g(a)o(v)m(ailable)i (system)d(bu\013er)i(space)f(is)h(limited)h({)e(a)g(safe)g(program)f(cannot)h (rely)189 1241 y(on)f(it.)146 1348 y(This)g(design)h(p)q(oin)o(t)g(has)f(the) h(follo)o(wing)f(implications:)131 1454 y(1.)22 b(A)11 b(t)o(yp)q(e)g(1)f(or) h(t)o(yp)q(e)g(2)g(sub)q(op)q(eration)g(will)i(alw)o(a)o(ys)d(complete,)i (indep)q(enden)o(tl)q(y)h(of)e(other)g(pro)q(cesses)189 1510 y(activities.)21 b(Of)15 b(course,)g(the)g(same)g(is)h(true)f(for)g(a)f(t)o (yp)q(e)i(4)f(sub)q(op)q(eration.)131 1604 y(2.)22 b(The)15 b(successful)h(completion)h(of)d(a)h(t)o(yp)q(e)g(3)g(sub)q(op)q(eration)h (ma)o(y)e(dep)q(end)j(on)e(the)g(participation)189 1661 y(of)h(other)g(pro)q (cesses.)24 b(A)17 b(send)g(ma)o(y)f(b)q(e)h(blo)q(c)o(k)o(ed)g(un)o(til)h(a) e(matc)o(hing)h(receiv)o(e)g(o)q(ccurs.)24 b(And,)17 b(of)189 1717 y(course,)h(a)g(receiv)o(e)h(ma)o(y)e(not)h(complete)g(un)o(til)i(a)d (matc)o(hing)h(send)h(o)q(ccurs.)29 b(\(This)18 b(extends)h(to)189 1774 y(collectiv)o(e)e(comm)o(unication.\))146 1880 y(T)l(o)e(this,)h(w)o(e)f (need)i(add)f(p)q(ositiv)o(e)g(requiremen)o(ts)g(for)f(progress)g(and/or)h (fairness.)21 b(Let's)16 b(sa)o(y)f(that)75 1936 y(a)k(comm)o(unication)h(is) f Fc(enable)n(d)f Fg(\(at)h(a)f(pro)q(cess\))h(if)h(a)f(matc)o(hing)g(comm)o (unication)h(is)f(activ)o(e)h(\(i.e.)31 b(a)75 1993 y(matc)o(hing)16 b(receiv)o(e)h(for)e(a)h(send)g(or)f(a)h(matc)o(hing)g(send)g(for)g(a)f (receiv)o(e\).)23 b(An)16 b(enabled)h(comm)o(unication)75 2049 y(ma)o(y)g(b)q(ecome)i(disabled)h(either)e(b)q(ecause)h(it)g(completes)f (successfully)i(\(the)e(comm)o(unication)h(o)q(ccurs)75 2106 y(and)d(sub)q(op)q(eration)h(3)e(terminates\))g(or)h(b)q(ecause)g(the)g(matc) o(hing)g(comm)o(unication)h(b)q(ecomes)f(inactiv)o(e)75 2162 y(\(e.g.,)e(a)h(receiv)o(e)h(that)e(matc)o(hes)h(one)h(send)g(ma)o(y)e(b)q(e) i(satis\014ed)g(b)o(y)f(another)g(send,)h(th)o(us)f(disabling)i(the)75 2219 y(\014rst)e(send\).)75 2325 y Ff(progress)21 b Fg(If)14 b(some)f(comm)o(unication)h(in)g(the)f(system)g(is)h(enabled)h(then)e(some)g (comm)o(unication)h(ev)o(en-)189 2381 y(tually)i(o)q(ccurs)f(in)h(the)g (system.)75 2475 y Ff(fairness)22 b Fg(An)c(activ)o(e)f(comm)o(unication)h (either)g(completes)g(successfully)h(or)d(b)q(ecomes)i(p)q(ermanen)o(tly)189 2532 y(disabled.)964 2828 y(5)p eop %%Page: 6 6 bop 146 45 a Fg(W)l(e)19 b(assume)g(in)h(these)f(de\014nitions)i(that)d(eac)o (h)h(pro)q(cess)g(in)o(v)o(ok)o(e)g(sub)q(op)q(erations)h(in)g(the)f(correct) 75 102 y(order,)c(i.e.)23 b(1\(23\))393 85 y Fe(\003)411 102 y Fg(4.)f(Of)16 b(course,)f(a)h(program)f(where)h(an)g(activ)o(e)g(comm)o (unication)g(b)q(ecomes)h(p)q(erma-)75 158 y(nen)o(tly)f(disabled)h(\(e.g.,)c (a)i(send)h(is)f(left)h(forev)o(er)e(with)i(no)f(matc)o(hing)g(receiv)o(e\))h (is)f(erroneous,)g(and)g(will)75 214 y(not)g(terminate)g(successfully)l(.)146 271 y(The)20 b(enforcemen)o(t)g(of)g(these)g(t)o(w)o(o)f(conditions)i (requires)g(a)f(\015o)o(w)g(con)o(trol)g(proto)q(col:)29 b(A)21 b(pro)q(cess)75 327 y(should)c(not)f(b)q(e)h(prev)o(en)o(ted)g(from)e (accepting)j(one)e(message)g(that)g(w)o(as)f(sen)o(t)h(b)q(ecause)i(its)e (bu\013ers)g(are)75 384 y(full)21 b(with)f(other)g(messages,)g(p)q(ossibly)h (sen)o(t)f(earlier.)34 b(When)21 b(a)e(receiv)o(e)i(matc)o(hes)e(sev)o(eral)h (p)q(ossible)75 440 y(messages)d(\(e.g.,)f(a)h(receiv)o(e)h(with)g(a)f (\\don't)f(care")h(sender)h(\014eld\))g(then)g(alternativ)o(e)g(options)f (should)75 497 y(b)q(e)f(considered)g(fairly)g(\(e.g.,)e(b)o(y)h(using)h(an)f (LR)o(U)h(p)q(olicy)h(when)e(considering)i(matc)o(hing)e(senders\).)146 553 y(A\014cionados)d(of)f(formalism)h(will)i(note)d(that)g(I)h(used)g(the)g (w)o(eak)o(est)f(de\014nition)j(of)d(fairness)h(\(ev)o(en)o(tual)75 610 y(fairness\),)j(rather)f(than)h(b)q(ounded)i(fairness)f(or)e(other)h (alternativ)o(es.)146 666 y(Examples:)146 723 y(The)g(follo)o(wing)h (programs)e(are)h(safe,)f(and)i(should)g(succeed:)75 823 y Fb(!)24 b(assume)f(only)g(two)h(processes,)e(numbered)h(1)h(and)f(2)75 936 y(IF)h(\(GETID\(\))e(==)i(1\))g(THEN)147 992 y(DO)f(I=0,)g(N)218 1048 y(CALL)g(CSEND\(mode=nonblocking,)e(buf=buf\(I\),)i(dest=2,)f(type=I,)h (len=1000\))147 1105 y(END)g(DO)75 1218 y(ELSE)g(!)h(\(GETID\(\))f(==)g(2\)) 147 1274 y(DO)g(I=0,)g(N)218 1331 y(CALL)g(CRECV\(mode=nonblocking,)e (buf=buf\(I\),)i(dest=2,)f(type=N-I,)h(len=1000\))147 1387 y(END)g(DO)146 1487 y Fg(Pro)q(cess)16 b(1)g(uses)h(non)o(blo)q(c)o(king)h (sends)f(to)f(send)h(a)f(sequence)i(of)e(messages)g(to)f(pro)q(cess)i(2;)g (pro)q(cess)75 1544 y(2)d(receiv)o(es)h(these)f(messages)f(in)i(the)f(rev)o (erse)g(order,)g(using)h(non)o(blo)q(c)o(king)g(receiv)o(es.)20 b(In)15 b(practice,)g(suc)o(h)75 1600 y(program)e(ma)o(y)g(fail)h(if)g(the)g (n)o(um)o(b)q(er)g(of)g(p)q(ending)h(messages)f(implied)i(b)o(y)d(this)i (program)d(is)i(larger)g(than)75 1657 y(the)20 b(system)g(limit.)36 b(But)21 b(the)f(system)g(limit)h(should)h(b)q(e)e(large)h(enough)f(as)g(to)g (mak)o(e)f(suc)o(h)i(failure)75 1713 y(extremely)16 b(unlik)o(ely)l(.)75 1813 y Fb(!)24 b(Assume)f(a)g(large)h(number)f(of)g(processes)75 1926 y(!)h(Consumer)e(code)75 1983 y(IF)i(\(GETID\(\))e(==)i(1\))g(THEN)123 2039 y(DO)f(I=)h(1,)f(INFOG\(\))194 2095 y(CALL)h(CRECV\(mode=blocking,)d (buf=buf\(I\),)h(source=I,)h(type=0,)g(len=1000\))123 2152 y(END)g(DO)75 2265 y(!)h(Producers)75 2321 y(ELSE)123 2378 y(CALL)f(CSEND\(mode=blocking,)e(buf=sendbuf,)h(dest=1,)h(type=0,)g (len=1000\))146 2478 y Fg(All)17 b(pro)q(cesses)f(send)g(a)f(message)h(to)f (pro)q(cess)h(1.)21 b(Pro)q(cess)15 b(1)h(receiv)o(es)g(these)g(messages)f (in)i(a)e(\014xed)75 2534 y(order.)31 b(A)20 b(\015o)o(w)e(con)o(trol)h (proto)q(col)g(is)h(needed)g(to)f(mak)o(e)f(sure)h(that)g(bu\013ers)g(at)f (pro)q(cess)i(1)f(will)h(not)75 2591 y(o)o(v)o(er\015o)o(w.)e(The)c(sending)h (pro)q(cesses)f(ma)o(y)g(b)q(e)g(dela)o(y)o(ed)h(un)o(til)g(the)f(receiv)o(e) g(pro)q(cess)g(is)h(ready)f(to)f(accept)75 2647 y(their)j(message.)146 2704 y(Consider)f(the)h(follo)o(wing)g(implemen)o(tation)g(alternativ)o(es)f (for)g(this)h(co)q(de:)964 2828 y(6)p eop %%Page: 7 7 bop 143 45 a Fa(\017)23 b Fg(Messages)18 b(are)h(sen)o(t)g(immediately)i(and) e(stored)g(in)h(the)f(ph)o(ysical)i(memory)d(of)h(the)g(receiving)189 102 y(no)q(de)g({)g(no)g(pro)o(vision)g(for)g(retry)l(.)31 b(This)19 b(is)h(a)e(bad)i(implemen)o(tation)g(since)g(ph)o(ysical)g(storage) 189 158 y(limits)c(are)f(lik)o(ely)i(to)e(b)q(e)g(encoun)o(tered.)143 252 y Fa(\017)23 b Fg(Messages)c(are)g(sen)o(t)h(immediately)h(and)f(stored)f (in)i(the)f(receiv)o(er)g(ph)o(ysical)i(memory)l(.)33 b(If)20 b(the)189 308 y(receiv)o(er)e(is)g(unable)h(to)f(receiv)o(e)g(the)g(message,) g(the)f(send)i(is)f(retried.)28 b(This)19 b(is)f(a)f(correct,)h(but)189 365 y(p)q(ossibly)e(ine\016cien)o(t)h(implemen)o(tation.)143 459 y Fa(\017)23 b Fg(A)16 b(\\request-to-send")h(proto)q(col)g(is)g(used;)h (the)f(receiv)o(er)g(allo)o(ws)g(the)g(send)g(to)g(pro)q(ceed)g(only)g(if)189 515 y(it)g(has)g(storage)f(for)h(the)g(message.)26 b(The)18 b(receiv)o(ers)g(need)g(to)e(store)h(information)g(on)h(p)q(ending,)189 571 y(unsatis\014ed)13 b(requests)g(to)f(send.)19 b(The)13 b(maxim)o(um)f(n)o(um)o(b)q(er)h(of)f(p)q(ending)j(requests)d(at)g(a)g(no)q (de)h(is)g(a)189 628 y(system)f(limit)i(that)e(should)i(b)q(e)f(set)f(large)h (enough)g(as)f(to)g(b)q(e)i(considered)g(\\practically)g(in\014nite".)146 722 y(Opinions?)146 778 y(Should)k(w)o(e)e(b)q(e)h(ev)o(en)h(more)e (restrictiv)o(e)h(and)g(require)g(that)g(safe)f(programs)f(run)i(correctly)l (,)h(ev)o(en)75 835 y(when)c(exceeding)h(\\v)o(ery)d(large")h(resource)h (limits?)21 b(\(Or)13 b(should)i(w)o(e)e(a)o(v)o(oid)g(to)f(burden)j(the)e (implemen-)75 891 y(tation)i(with)g(requiremen)o(ts)h(to)f(handle)h(what)f (most)f(lik)o(ely)j(are)e(pathologic)h(programs?\))146 948 y(Should)h(w)o(e)f(try)g(to)g(b)q(e)h(more)f(formal)g(in)h(de\014ning)h(when) f(some)f(of)g(the)h(\\unsafe")f(programs)f(will)75 1004 y(run)h(to)g (completions?)24 b(\(Can)16 b(w)o(e)g(do)g(it)g(without)g(b)q(eing)i(more)d (sp)q(eci\014c)j(ab)q(out)e(the)h(implemen)o(tation)75 1060 y(mo)q(del?\))146 1117 y(Should)f(w)o(e)f(require)h(fairness?)21 b(Should)c(w)o(e)d(ha)o(v)o(e)h(a)g(stronger)f(requiremen)o(t)i(of)f (fairness?)75 1260 y Fh(What)23 b(are)g(the)g(comm)n(unication)d(parameters?) 75 1363 y Fd(Message)e(data)i(\(bu\013er\))131 1449 y Fg(1.)i(scalar)15 b(\(b)o(yte/halfw)o(ord/w)o(ord/doublew)o(ord\),)e(presumably)j(coming)f (from)g(a)g(register)131 1543 y(2.)22 b(Con)o(tiguous)15 b(space)g(in)h (memory)f(\(initial)i(address/length\))131 1637 y(3.)22 b(Bu\013er)15 b(with)g(constan)o(t)g(stride)g(\(initial)i(address/size)f(of)f(eac)o(h)g (blo)q(c)o(k/stride/n)o(um)o(b)q(er)i(blo)q(c)o(ks\))131 1731 y(4.)22 b(A)15 b(t)o(yp)q(ed)g(message)g(\(of)g(a)f(t)o(yp)q(e)i(supp)q (orted)g(in)g(the)f(domain)g(language\).)131 1824 y(5.)22 b(A)d(union)i(of)d (sev)o(eral)i(messages)f(\(p)q(ossibly)i(sp)q(eci\014ed)g(b)o(y)e(an)h(arra)o (y)e(of)h(p)q(oin)o(ters)h(to)e(message)189 1881 y(descriptors\).)g(The)13 b(seman)o(tics)f(of)f(a)h(union)h(is)f(probably)h(conjunctiv)o(e)f(at)g(the)g (sender)g(end)h(\(send)189 1937 y(all\);)g(at)f(the)h(receiv)o(e)g(end,)h(it) f(ma)o(y)e(b)q(e)j(conjunctiv)o(e)f(\(receiv)o(e)g(all\))h(or)e(disjunctiv)o (e)i(\(receiv)o(e)f(an)o(y\).)146 2044 y(The)j(curren)o(t)g(MPI)f(prop)q (osal)i(has)e(2)h(and)g(3)g(\(2)f(is)h(really)h(a)f(particular)g(case)g(of)g (3\).)21 b(Do)15 b(w)o(e)h(w)o(an)o(t)75 2100 y(more?)k(In)c(particular,)g (do)f(w)o(e)g(w)o(an)o(t)f(t)o(yp)q(ed)h(messages?)20 b(Ho)o(w)15 b(general?)75 2222 y Fd(Groups)k(and)g(con)n(text)75 2307 y Fg(Messages)c(are)g(tagged)g(b)o(y)g(pro)q(cess)h(names)f(\(sender)h(and)f (receiv)o(er\))h(and)g(t)o(yp)q(e)f(names.)21 b(Both)15 b(names)75 2364 y(spaces)g(can)f(b)q(e)i(\015at)e(or)g(structured.)20 b(If)14 b(a)h(\015at)f(pro)q(cess)h(name)f(space)h(is)g(used,)g(then)g(a)f (pro)q(cess)h(names)75 2420 y(is)e(an)f(in)o(teger)g(\(from)f(0)h(to)f(n)o (um)o(b)q(er)p 687 2420 14 2 v 17 w(of)p 741 2420 V 16 w(pro)q(cesses-1,)i (for)e(C)h(think)o(ers\).)19 b(If)13 b(a)e(structured)i(pro)q(cess)f(name)75 2477 y(space)18 b(is)g(used)f(then)h(a)f(pro)q(cess)h(name)f(is)h(a)f(pair)h (of)f(the)g(form)g(\(group,)g(rank\).)26 b(Similarly)19 b(in)f(a)f(\015at)75 2533 y(t)o(yp)q(e)g(name)h(space,)g(a)f(t)o(yp)q(e)g(is)h(a)f(n)o(um)o(b)q (er)h(in)g(a)f(range;)h(in)g(a)f(structured)g(t)o(yp)q(e)h(name)f(space,)h (it)f(is)h(a)75 2590 y(pair)e(of)e(the)i(form)e(\(group,)g(lo)q(cal)p 659 2590 V 18 w(t)o(yp)q(e\).)146 2646 y(The)21 b(usefulness)h(of)f(groups)f (seem)h(clear)h(in)f(the)g(con)o(text)g(of)f(collectiv)o(e)j(op)q(erations,)f (where)f(a)75 2703 y(groupid)c(is)g(a)f(handle)h(to)f(sp)q(ecify)h(the)g(set) f(of)f(participan)o(ts)i(in)g(a)f(collectiv)o(e)i(comm)o(unication)f(\(Note:) 964 2828 y(7)p eop %%Page: 8 8 bop 75 45 a Fg(some)15 b(of)h(the)f(discussions)j(on)d(groups)g(seem)h(to)f (assume)h(that)f(a)g(group)g(is)i(alw)o(a)o(ys)e(represen)o(ted)h(b)o(y)f(a) 75 102 y(mem)o(b)q(ership)f(list)g(that)f(is)h(presen)o(t)f(at)f(eac)o(h)i (group)f(mem)o(b)q(er.)19 b(While)14 b(is)g(the)f(represen)o(tation)h(lik)o (ely)h(to)75 158 y(b)q(e)g(used)h(on)e(small)i(systems,)e(it)h(is)g(b)o(y)g (no)g(means)f(the)h(unique)h(p)q(ossible)h(one.)j(An)o(y)15 b(connected)g(graph)75 214 y(structure)h(can)h(b)q(e)g(used)g(to)f(k)o(eep)h (trac)o(k)f(of)g(group)g(mem)o(b)q(ership,)i(with)f(ob)o(vious)f(tradeo\013s) g(b)q(et)o(w)o(een)75 271 y(storage)h(and)h(p)q(erformance.)28 b(A)18 b(complete)g(list)h(at)e(eac)o(h)h(no)q(de)h(is)f(one)g(extreme)g(of)g (this,)g(namely)l(,)h(a)75 327 y(complete)g(graph.)28 b(Therefore,)19 b(a)e(group)h(handle)i(is)e(a)g(more)g(abstract)f(and)h(more)g(general)g (concept)75 384 y(than)d(a)g(list)h(of)f(group)g(mem)o(b)q(ers.\))146 440 y(The)h(collectiv)o(e)j(comm)o(unication)e(sub)q(committee)g(has)g(to)f (deal)h(with)g(mec)o(hanisms)g(for)f(creating)75 497 y(and)j(destro)o(ying)f (groups)g(\(for)g(the)g(record,)h(I)g(fa)o(v)o(or)e(dynamic)i(group)g (creation)f(b)q(oth)h(b)o(y)f(explicitly)75 553 y(listing)j(group)f(mem)o(b)q (ers)f(and)h(b)o(y)g(partitioning)h(an)e(existing)i(group.)33 b(I)20 b(do)g(not)f(fa)o(v)o(or)g(a)g(complex)75 610 y(stac)o(k)e(or)h(tree)g (mec)o(hanism)h(to)e(manage)h(groups)f(and)i(na)o(vigate)e(from)h(one)g (group)g(to)f(another)h({)g(at)75 666 y(least)d(not)g(in)h(the)f(basic)h(MPI) f(prop)q(osal\).)146 723 y(In)i(the)f(con)o(text)g(of)g(p)q(oin)o(t-to-p)q (oin)o(t)h(comm)o(unication,)g(the)f(issue)h(is)g(whether)g(w)o(e)f(w)o(an)o (t)f(to)h(use)g(a)75 779 y(structured)f(name)g(space)h(for)e(t)o(yp)q(e)i (and)f(pro)q(cess)h(argumen)o(ts)e(and,)h(if)h(so,)e(ho)o(w.)146 835 y(The)21 b(adv)m(an)o(tage)h(of)f(ha)o(ving)h(a)f(structured)h(name)g (space)f(for)h(pro)q(cesses)g(is)g(that)f(it)h(simpli\014es)75 892 y(com)o(bining)14 b(of)e(distinct)i(parallel)g(co)q(des)f(in)h(a)e(lo)q (osely)i(coupled)g(application:)20 b(Eac)o(h)13 b(functional)h(subset)75 948 y(can)j(use)h(message)f(passing)h(within)g(its)g(con)o(text,)f(with)g(no) h(c)o(hanges)f(in)h(the)f(lo)q(cal)i(co)q(de.)27 b(The)17 b(same)75 1005 y(concern)k(for)f(mo)q(dularit)o(y)h(militates)h(for)e(a)g(structured)g (t)o(yp)q(e)h(space.)36 b(This)21 b(do)q(es)g(not)g(necessarily)75 1061 y(implies)j(or)d(requires)h(a)f(stac)o(k)g(mec)o(hanism)h(for)f (managing)g(groups,)i(or)e(a)g(tree)g(structure)h(among)75 1118 y(groups.)146 1174 y(First,)14 b(t)o(w)o(o)g(remarks:)143 1268 y Fa(\017)23 b Fg(The)c(preexisting)h(group)f Fb(ALL)f Fg(that)g(includes)j(all)f(pro)q(cesses)g(pro)o(vides)f(a)g(global)g(con)o (text.)31 b(If)189 1324 y(all)17 b(comm)o(unications)f(use)h(this)f(con)o (text,)f(then)i(one)f(defaults)g(to)g(\015at)f(name)h(space.)23 b(With)16 b(the)189 1381 y(righ)o(t)g(syn)o(tax,)h(the)g(existence)h(of)f(a)g (structured)g(name)g(space)g(can)g(b)q(e)h(ignored)g(b)o(y)f(users)g(that)189 1437 y(prefer)e(a)g(\015at,)f(global)i(name)f(space.)143 1531 y Fa(\017)23 b Fg(A)c(structured)f(name)h(space)g(can)g(alw)o(a)o(ys)g(fak)o (ed,)g(with)g(no)g(c)o(hanges)g(in)g(the)g(comm)o(unication)189 1588 y(op)q(erations:)g(Replace)d(eac)o(h)e Fb(process)p 868 1588 15 2 v 16 w(id)g Fg(actual)h(parameter)e(b)o(y)h(a)g(a)g(reference)h(to) e(a)h(function)189 1644 y Fb(process\(rank,)22 b(group)p 646 1644 V 16 w(id\))p Fg(;)c(replace)h(eac)o(h)e Fb(type)h Fg(actual)f (parameter)g(b)o(y)h(a)f(reference)h(to)f(a)189 1701 y(function)f Fb(type\(local)p 610 1701 V 15 w(type,)24 b(group)p 889 1701 V 16 w(id\))p Fg(.)146 1794 y(F)l(or)16 b(simplicit)o(y)l(,)j(I)e(shall)h (assume)f(the)f(same)h(group)f(con)o(text)h(is)g(used)g(b)q(oth)g(for)f (relativ)o(e)i(pro)q(cess)75 1851 y(n)o(um)o(b)q(ering)h(and)g(for)f(relativ) o(e)h(t)o(yp)q(e)g(naming.)30 b(I)19 b(b)q(eliev)o(e)h(this)f(simpli\014es)i (the)d(discussion,)j(without)75 1907 y(in)o(tro)q(ducing)13 b(undue)g(restrictions.)20 b(A)12 b(con)o(text)f(for)g(a)h(comm)o(unication)g (op)q(eration)h(can)f(b)q(e)g(established)75 1964 y(in)k(t)o(w)o(o)e(w)o(a)o (ys:)131 2070 y(1.)22 b(The)f(curren)o(t)h(scop)q(e)g(\(curren)o(t)f(group\)) g(of)g(a)g(comm)o(unication)h(op)q(eration)g(is)g(implicit;)27 b(it)21 b(is)189 2126 y(established)16 b(b)o(y)g(the)f(last)g(executed)h(con) o(text)f(con)o(trolling)h(call)g(\()p Fb(MPI)p 1401 2126 V 17 w(PUSHC)e Fg(and)i Fb(MPI)p 1713 2126 V 16 w(POPC)f Fg(in)189 2183 y(the)e(curren)o(t)h(draft\).)k(Th)o(us,)c(an)o(y)f(pro)q(cess)h(n)o(um) o(b)q(er)h(is)f(understo)q(o)q(d)g(to)f(b)q(e)i(relativ)o(e)f(rank)f(in)i (the)189 2239 y(curren)o(t)g(group,)f(and)h(an)o(y)g(t)o(yp)q(e)g(name)g(is)h (understo)q(o)q(d)f(to)g(b)q(e)g(relativ)o(e)h(to)e(curren)o(t)h(group,)g (and)189 2296 y(do)q(es)j(not)f(con\015icts)i(with)f(t)o(yp)q(e)h(names)e (used)i(within)g(other)f(groups.)27 b(Of)18 b(course,)h(the)f(initial)189 2352 y(scop)q(e)d(is)g Fb(ALL)p Fg(,)e(and)i(the)g(system)f(defaults)h(to)f (a)g(\015at)g(name)h(space)g(if)g(scop)q(e)g(setting)f(commands)189 2409 y(are)h(nev)o(er)g(used.)131 2503 y(2.)22 b(The)g(con)o(text)g(of)f(a)h (comm)o(unication)h(op)q(erations)f(need)h(to)f(b)q(e)h(explicitly)i(sp)q (eci\014ed)f(in)f(the)189 2559 y(comm)o(unication)15 b(op)q(eration)f (itself.)21 b(A)14 b(pro)q(cess)h(ma)o(y)e(b)q(elong)j(to)d(sev)o(eral)i (groups)f(at)f(an)o(y)h(p)q(oin)o(t)189 2615 y(in)19 b(time,)g(and)g(ma)o(y)f (use)h(an)o(y)f(of)g(these)h(groups)f(as)g(scop)q(e)h(for)f(a)g(comm)o (unication)h(op)q(eration.)189 2672 y(Again,)14 b(the)f(initial)j(scop)q(e)e Fb(ALL)f Fg(is)h(prede\014ned,)h(and)f(can)g(b)q(e)g(used)g(as)g(a)f(\015at)g (name)h(space.)19 b(\(In)14 b(a)964 2828 y(8)p eop %%Page: 9 9 bop 189 45 a Fg(language)13 b(where)g(argumen)o(ts)f(are)h(optional,)h(con)o (text)e(w)o(ould)i(b)q(e)g(an)f(optional)g(parameter)g(with)189 102 y(default)i(v)m(alue)i Fb(ALL)p Fg(.\))146 208 y(The)h(adv)m(an)o(tage)f (of)g(an)h(explicit)i(con)o(text)d(parameter)g(is)h(that)g(it)g(prev)o(en)o (ts)f(the)h(confusion)h(that)75 264 y(o)q(ccurs)13 b(when)g(an)g(implicit)i (con)o(text)d(is)h(c)o(hanged)g(mid)g(comm)o(unication.)20 b(More)12 b(generally)l(,)i(it)f(prev)o(en)o(ts)75 321 y(the)e(confusion)h (arising)g(when)g(the)f(seman)o(tics)h(of)e(an)i(op)q(eration)f(is)h (quali\014ed)h(b)o(y)e(a)g(scop)q(e)h(that)e(dep)q(ends)75 377 y(on)18 b(the)g(con)o(trol)g(\015o)o(w)g(of)g(the)g(program,)g(rather)f (than)h(its)h(static)f(structure.)29 b(The)18 b(disadv)m(an)o(tage)g(is)75 434 y(the)e(need)g(for)f(an)g(additional)i(parameter,)d(in)j(languages)e (that)g(don't)g(ha)o(v)o(e)g(optional)h(parameters)e(or)75 490 y(o)o(v)o(erloading)h(of)g(pro)q(cedures.)146 547 y(Opinions?)75 690 y Fh(Matc)n(hing)23 b(of)g(send)g(and)h(receiv)n(e)75 793 y Fd(receiv)n(e)18 b(criteria)75 879 y Fg(Messages)c(are)h(matc)o(hed)g(b)o (y)h(sender)f(and)h(t)o(yp)q(e,)f(where)g(eac)o(h)g(can)h(b)q(e)g(a)e (\\don't)h(care".)146 935 y(Our)f(exp)q(erience)j(is)e(that)f(matc)o(hing)g (b)o(y)g(t)o(yp)q(e)h(only)l(,)g(or)f(disallo)o(wing)i(\\don't)e(care")g(in)h (the)f(sender)75 992 y(\014eld)20 b(signi\014can)o(tly)h(simpli\014es)g(the)e (implemen)o(tation)i({)d(but)h(some)g(users)g(complained)i(ab)q(out)e(suc)o (h)75 1048 y(restriction.)146 1105 y(Also,)c(do)g(w)o(e)g(w)o(an)o(t)f(to)g (b)q(e)i(explicit)i(ab)q(out)d(the)g(don't)g(care)g(v)m(alue,)h(or)e(use)i (named)f(constan)o(ts?)146 1161 y(Opinions?)75 1283 y Fd(Matc)n(hing)20 b(of)f(data)75 1369 y Fg(F)l(or)c(un)o(t)o(yp)q(ed)g(messages)g(the)g (options)h(are)131 1462 y(1.)22 b(Send)16 b(and)f(receiv)o(e)h(bu\013er)f(ha) o(v)o(e)g(same)g(size)131 1556 y(2.)22 b(Receiv)o(e)16 b(bu\013er)g(is)f(no)g (shorter)g(than)g(send)h(bu\013er)131 1650 y(3.)22 b(No)15 b(constrain)o(ts)f(\(messages)h(ma)o(y)f(b)q(e)i(truncated\).)131 1744 y(4.)22 b(User)15 b(c)o(ho)q(ose)g(one)g(of)g(the)g(ab)q(o)o(v)o(e)g (options)h(\(one)f(more)f(call)j(parameter,)d(or)g(a)h(global)h(\015ag\).)146 1838 y(Choices)f(should)g(b)q(e)g(the)f(same)g(for)f(all)j(t)o(yp)q(es)e(of)g (send)h(op)q(eration)f(\(the)g(curren)o(t)g(MPI)g(draft)g(seem)75 1894 y(to)h(allo)o(w)g(truncation)g(for)g(con)o(tiguous)g(data,)f(but)i (disallo)o(w)g(it)f(for)g(receiv)o(e)h(with)g(stride\))146 1951 y(The)d(EUI)h(prop)q(osal)f(uses)g(option)h(4.)19 b(I)13 b(w)o(ould)h(b)q(e)g(happier)g(with)f(2,)g(but)h(am)e(uncomfortable)i(with)75 2007 y(3)i(\(truncation)h(seems)g(to)f(fall)i(in)f(the)g(category)f(of)g (comm)o(unication)i(errors,)e(for)g(whic)o(h)h(I)h(assume)e(a)75 2063 y(global)g(exception)g(handling)h(mec)o(hanism\).)146 2120 y(Opinions?)146 2176 y(If)12 b(t)o(yp)q(ed)g(messages)g(are)g(supp)q (orted)g(then)h(one)f(should)h(sp)q(ell)h(out)e(requiremen)o(ts)g(on)g(t)o (yp)q(e)g(compat-)75 2233 y(ibilit)o(y)18 b(\(e.g.,)d(can)h(t)o(w)o(o)e(real) i(n)o(um)o(b)q(ers)g(b)q(e)h(receiv)o(ed)g(in)o(to)f(a)g(complex)g(n)o(um)o (b)q(er?\))23 b({)16 b(this)g(is)g(probably)75 2289 y(sp)q(eci\014c)h(to)e (eac)o(h)g(language)g(binding.)146 2346 y(Opinions?)75 2489 y Fh(Syn)n(tax)75 2590 y Fg(This)h(ma)o(y)e(b)q(e,)i(of)f(course,)f(a)h(sub)s (ject)g(of)g(endless)i(dispute.)k(A)15 b(few)g(ob)o(vious)g(guidelines:)143 2684 y Fa(\017)23 b Fg(Short,)14 b(mnemonic)i(names)964 2828 y(9)p eop %%Page: 10 10 bop 143 45 a Fa(\017)23 b Fg(Systematic)15 b(naming)h(con)o(v)o(en)o(tion)143 137 y Fa(\017)23 b Fg(Systematic)15 b(ordering)g(of)g(parameters)143 230 y Fa(\017)23 b Fg(Use,)15 b(in)h(languages)g(where)g(this)f(is)h(p)q (ossible,)h(of)e(optional)h(parameters)f(and)h(k)o(eyw)o(ord)e(param-)189 286 y(eters.)143 379 y Fa(\017)23 b Fg(Use,)15 b(in)h(languages)f(where)g (this)h(p)q(ossible,)h(of)d(o)o(v)o(erloading.)146 469 y(I)d(heard)g(in)h (our)f(last)g(meeting)g(a)g(preference)h(for)f(m)o(ultiple)h(function)g (names,)g(rather)e(than)h(m)o(ultiple)75 525 y(mo)q(des.)146 582 y(Th)o(us,)f(in)h(C++)g(or)f(F90,)f(one)i(could)g(ha)o(v)o(e)f(the)g(c)o (hoice)h(of)f(the)g(t)o(yp)q(e)g(of)g(datum)g(sen)o(t)g(\(scalar/con)o (tiguous)75 638 y(v)o(ector/noncon)o(tiguous\))15 b(b)q(e)i(dictated)g(b)o(y) f(the)h(t)o(yp)q(e)f(and)h(n)o(um)o(b)q(er)f(of)g(parameters,)f(rather)h (than)g(b)q(e)75 695 y(enco)q(ded)f(in)f(the)g(function)g(name.)19 b(Then)14 b(a)f(p)q(ossible)i(naming)f(c)o(hoice)h(is)f Fb(mode)23 b(|)h(operation)p Fg(,)12 b(where)75 751 y Fb(mode)18 b Fg(=)g Fb(init)g Fg(\(1\),)f Fb(nonblocking)g Fg(\(1+2\),)h Fb(blocking)f Fg(\(1+2+3+4,)h(lo)q(cal)h(termination\),)g(or)f Fb(sync)75 808 y Fg(\(1+2+3+4)d(global)h(termination\),)f(and)g Fb(operation)f Fg(=)h(send,)h(or)e(recv.)146 864 y(In)h(addition,)h(one)f(has)g Fb(DO\(handle\))p Fg(,)f Fb(START\(handle\))p Fg(,)e Fb(AWAIT\(handle\))i Fg(and)h Fb(FREE\(handle\))p Fg(.)146 921 y(Another)21 b(issue)i(is)f (function)g(vs)f(pro)q(cedure)i(call,)h(in)e(F)l(ortran.)38 b(Pro)q(cedure)22 b(calls)h(seem)e(more)75 977 y(natural)15 b(in)h(F)l(ortran,)e(where)h(function)h(in)o(v)o(o)q(cations)g(cannot)f(b)q (e)h(used)g(as)e(statemen)o(ts.)146 1033 y(Opinions?)22 b(Preferences)16 b(for)f(mnemonics?)21 b(Suggestions)16 b(for)e(F77)h(and)g(C?)75 1176 y Fh(Miscelania)75 1277 y Fg(Do)h(w)o(e)g(w)o(an)o(t)g(a)g Fb(TEST)p Fg(,)f(a)i(non)o(blo)q(c)o(king)h Fb(AWAIT)d Fg(op)q(eration?)25 b(\(Alw)o(a)o(ys)16 b(returns;)h(a)f(successful)i(return)75 1334 y(is)e(equiv)m(alen)o(t)h(to)d(a)h(successful)i(return)e(of)g(a)f(blo)q (c)o(king)j Fb(AWAIT)p Fg(\).)146 1390 y(Do)d(w)o(e)h(w)o(an)o(t)f(to)h Fb(AWAIT)f Fg(sev)o(eral)i(messages?)146 1447 y(Do)h(w)o(e)h(w)o(an)o(t)f(a)g Fb(PROBE)h Fg(function)h(\(tells)f(me)g(whic)o(h)h(messages)f(are)g(w)o (aiting)g(to)f(b)q(e)i(receiv)o(ed)g(b)o(y)75 1503 y(me\)?)146 1560 y(Where)12 b(do)h(w)o(e)f(return)h(information)g(on)g(parameters)f(of)g (receiv)o(ed)i(messages)e(suc)o(h)h(as)f(length,)i(and)75 1616 y(t)o(yp)q(e?)24 b(The)16 b(curren)o(t)g(MPI)h(prop)q(osal)f(has)g(length)h (b)q(e)g(the)f(v)m(alue)i(returned)e(b)o(y)h(the)f(receiv)o(e)h(function)75 1673 y(and)d(t)o(yp)q(e)g(returned)h(b)o(y)f(a)f(subsequen)o(t)i(call)g(to)e Fb(MP)p 964 1673 15 2 v 17 w(INFOT)p Fg(.)g(Another)h(natural)g(c)o(hoice)h (w)o(ould)f(seem)h(to)75 1729 y(b)q(e)f(that)f(that)f(length)i(and)g(t)o(yp)q (e)f(are)g(returned)h(b)o(y)f(the)h(len)g(and)g(t)o(yp)q(e)f(parameters)f(of) h(the)h(initial)h(call,)75 1786 y(but)g(this)h(is)g(error)e(prone:)143 1876 y Fa(\017)23 b Fg(The)d(v)m(alues)h(are)e(returned)i(async)o(hronously)f (for)f(non)o(blo)q(c)o(king)j(receiv)o(es.)35 b(The)20 b(user)g(has)g(to)189 1932 y(remem)o(b)q(er)11 b(not)h(to)f(touc)o(h)g(the)h(actual)g(len)g(and)g (t)o(yp)q(e)g(parameters,)f(in)h(addition)h(of)e(not)h(touc)o(hing)189 1989 y(the)j(receiv)o(e)h(bu\013er,)f(un)o(til)h(the)f(op)q(eration)h (completes.)143 2081 y Fa(\017)23 b Fg(A)f(natural)h(implemen)o(tation)h(w)o (ould)f(b)q(e)g(to)f(alw)o(a)o(ys)g(return)g(the)h(length)g(and)g(t)o(yp)q(e) g(of)f(the)189 2138 y(receiv)o(ed)f(messages.)36 b(Ho)o(w)o(ev)o(er,)20 b(most)g(users)g(will)i(assume)f(that)e(the)i(actual)g(len)g(and)g(t)o(yp)q (e)189 2194 y(parameters)14 b(are)h(not)g(up)q(dated)h(when)f(their)h(v)m (alue)h(is)e(not)g(\\don't)f(care".)143 2286 y Fa(\017)23 b Fg(The)13 b(user)h(will)h(not)e(b)q(e)h(able)g(to)f(use)h(constan)o(ts)e(as)h (actual)h(\\don't)f(care")g(parameter)f(v)m(alues)j(\(or)189 2343 y(an)o(ytime,)g(if)g(the)g(\\natural")g(implemen)o(tation)i(is)e(allo)o (w)o(ed\))143 2435 y Fa(\017)23 b Fg(This)17 b(c)o(hoice)h(b)q(ecomes)g(ev)o (en)f(less)h(app)q(etizing)h(when)e(a)g(handle)h(is)g(reused;)g(then)g(the)f (len)h(and)189 2492 y(t)o(yp)q(e)d(v)m(ariables)h(w)o(ould)g(b)q(e)g(reused,) f(to)q(o.)75 2582 y(It)g(seems)g(safer)f(to)g(return)h(these)g(v)m(alues)h (with)f(the)g(call)g(that)g(concludes)h(the)f(execution)h(of)e(sub)q(op)q (er-)75 2638 y(ation)h(3)g({)g(but)g(w)o(e)g(need)h(to)f(think)h(syn)o(tax.) 146 2695 y(Opinions?)952 2828 y(10)p eop %%Trailer end userdict /end-hook known{end-hook}if %%EOF From owner-mpi-comm@CS.UTK.EDU Mon Nov 30 18:08:53 1992 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA22099; Mon, 30 Nov 92 18:08:53 -0500 Received: by CS.UTK.EDU (5.61++/2.8s-UTK) id AA03436; Mon, 30 Nov 92 17:54:53 -0500 Received: from THUD.CS.UTK.EDU by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA03434; Mon, 30 Nov 92 17:54:51 -0500 From: Jack Dongarra Received: by thud.cs.utk.edu (5.61++/2.7c-UTK) id AA26470; Mon, 30 Nov 92 17:54:50 -0500 Date: Mon, 30 Nov 92 17:54:50 -0500 Message-Id: <9211302254.AA26470@thud.cs.utk.edu> To: mpi-comm@cs.utk.edu Subject: draft of introduction Enclosed is a draft the introduction subcommittee has put together. We would like to get your comments and input. We feel that it is in a partial state and will be expanded. Hopefully this draft will help keep the discussion of goals, target, audience, etc. from becoming fragmented in each subcommittee. Jack \section{Introduction} \label{sec:intro} \subsection{Overview and Goals} Message passing is a paradigm used widely on certain classes of parallel machines; especially those with distributed memory. Although there are many variations, the basic concept of processes communicating through messages is well understood. Over the last ten years, substantial progress has been made in casting significant applications in this paradigm. Each vendor has implemented their own variant. More recently, several systems \cite{need references} have demonstrated that a message passing system can be efficiently and be portably implemented. It is thus an appropriate time to try to define both the syntax and semantics of a core of library routines that will be useful to a wide range of users and efficiently implementable on a wide range of computers. In designing MPI we have sought to make use of the most attractive features of a number of existing message passing systems, rather than selecting one of them and adopting it as the standard. Thus, MPI has been strongly influenced by work at the IBM T. J. Watson Research Center by Bala, Kipnis, Snir and colleagues \cite{}, Intel's NX/2 \cite{}, Express \cite{}, nCUBE's Vertex \cite{}, and PARMACS \cite{}. Other important contributions have come from Zipcode \cite{}, Chimp \cite{}, PVM \cite{}, and PICL \cite{}. One of the objectives of this paper is to promote a discussion within the concurrent computing research community of the issues that must be addressed in establishing a practical, portable, and flexible standard for message passing. This cooperative process began with a workshop on standards for message passing held in April 1992 \cite{}. The main advantages of establishing a message passing standard are portability and ease-of-use. In a distributed memory communication environment in which the higher level routines and/or abstractions are build upon lower level message passing routines the benefits of standardization are particularly apparent. Furthermore, the definition of a message passing standard, such as that proposed here, provides vendors with a clearly defined base set of routines that they can implement efficiently, or in some cases provide hardware support for, thereby enhancing scalability. The goal of the Message Passing Interface simply stated is to develop a \it{de facto} standard for writing message-passing programs. As such the interface should establishing a practical, portable, efficient, and flexible standard for message passing. \subsection{Who Should Use This Standard?} This standard is intended for use by all those who want to write portable message-passing programs in Fortran 77 and/or C. This includes individual application programmers, developers software designed to run on parallel machines, and creators of higher-level programming languages, environments, and tools. In order to be attractive to this wide audience, the standard must provide a simple, easy-to-use interface for the basic user while not semantically precluding the high-performance message-passing operations available on advanced machines. \subsection{What Platforms Are Targets For Implementation?} The attractiveness of the message-passing paradigm at least partially stems from its wide portability. Programs expressed this way can run efficiently on distributed-memory multiprocessors, networks of workstations, and combinations of all of these. In addition, shared-memory implementations are possible. The paradigm will not be made obsolete by architectures combining the shared- and distributed-memory views, or by increases in network speeds. It thus should be both possible and useful to implement this standard on a great variety of machines, including those "machines" consisting of collections of other machines, parallel or not, connected by a communication network. \subsection{What Is Included In The Standard?} The standard includes (this is temporarily as inclusive as possible): \begin{itemize} \item Point-to-point communication in a variety of modes, including modes that allow fast communication and heterogeneous communication \item Collective operations \item Process groups \item Communication contexts \item A simple way to create processes for the SPMD model \item Bindings for both Fortran and C \item A model implementation \item A formal specification. \end{itemize} \subsection{What Is Not Included In The Standard?} The standard does not specify: \begin{itemize} \item Explicit shared-memory operations \item Operations that require more operating system support than is currently standard; for example, interrupt-driven receives, remote execution, or active messages \item Program construction tools \item Debugging facilities \item Tracing facilities \end{itemize} Features that are not included can always be offered as extensions by specific implementations. From owner-mpi-comm@CS.UTK.EDU Tue Dec 1 16:39:05 1992 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA17944; Tue, 1 Dec 92 16:39:05 -0500 Received: by CS.UTK.EDU (5.61++/2.8s-UTK) id AA27367; Tue, 1 Dec 92 16:14:27 -0500 Received: from relay2.UU.NET by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA27363; Tue, 1 Dec 92 16:14:24 -0500 Received: from uunet.uu.net (via LOCALHOST.UU.NET) by relay2.UU.NET with SMTP (5.61/UUNET-internet-primary) id AA00550; Tue, 1 Dec 92 16:14:28 -0500 Received: from kailand.UUCP by uunet.uu.net with UUCP/RMAIL (queueing-rmail) id 161350.23711; Tue, 1 Dec 1992 16:13:50 EST Received: from brisk.kai.com (brisk) by kailand.kai.com via SMTP (5.65d-92031301 for ) id AA22501; Tue, 1 Dec 1992 15:07:53 -0600 Received: by brisk.kai.com (920330.SGI-92101201) id AA16316; Tue, 1 Dec 92 15:07:51 -0600 Date: Tue, 1 Dec 92 15:07:51 -0600 Message-Id: <9212012107.AA16316@brisk.kai.com> To: mpi-comm@cs.utk.edu Subject: A proposal. From: Steven Ericsson Zenith Sender: zenith@kai.com Organization: Kuck and Associates, Inc. 1906 Fox Drive, Champaign IL USA 61820-7334, voice 217-356-2288, fax 217-356-5199 I've seen several proposals and made a few suggestions myself. I have now had a chance to look at Gropp and Lusk's Manual for the test implementation and I like it as a starting point. I propose to all subcommittees that we use this document as the focus for discussion along with the introduction Jack sent around yesterday. This shouldn't be taken as an endorsement by me of the Gropp and Lusk manual - I have several remarks I would like to make at a later stage. However, it is clear that we need a well considered and simple starting point and I see that in this document plus the introduction. Do I have any seconders? In the HPFF committe structure description we received a few days ago it was noted that much success came from someone taking charge of the central document. I implore someone to do this (Jack?). Steven P.S. I am not ignoring the other comments made in the committees - I think they too can - after debate - be folded into the suggested starting point. From owner-mpi-comm@CS.UTK.EDU Tue Dec 1 19:08:43 1992 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA21233; Tue, 1 Dec 92 19:08:43 -0500 Received: by CS.UTK.EDU (5.61++/2.8s-UTK) id AA01862; Tue, 1 Dec 92 18:43:11 -0500 Received: from cunyvm.cuny.edu by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA01858; Tue, 1 Dec 92 18:43:08 -0500 Message-Id: <9212012343.AA01858@CS.UTK.EDU> Received: from YKTVMV by CUNYVM.CUNY.EDU (IBM VM SMTP V2R2) with BSMTP id 7311; Tue, 01 Dec 92 18:42:33 EST Date: Tue, 1 Dec 92 18:43:02 EST From: "Marc Snir" X-Addr: (914) 945-3204 (862-3204) 28-226 IBM T.J. Watson Research Center P.O. Box 218 Yorktown Heights NY 10598 To: mpi-comm@cs.utk.edu Subject: HPF process Reply-To: SNIR@watson.ibm.com The HPF process did not start with a central document. Various peoples wrote proposals for various parts of the language design and, after several iterations, when the number of proposals starting being reasonable and these proposals started being coherent, they were collated into one document that evolved (mostly by successive pruning) into a draft. I suppose we can follow the same approach -- with the initial MPI Draft being our first frame of reference. From owner-mpi-comm@CS.UTK.EDU Fri Dec 4 16:10:12 1992 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA06597; Fri, 4 Dec 92 16:10:12 -0500 Received: by CS.UTK.EDU (5.61++/2.8s-UTK) id AA27740; Fri, 4 Dec 92 15:56:38 -0500 Received: from THUD.CS.UTK.EDU by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA27738; Fri, 4 Dec 92 15:56:36 -0500 From: Jack Dongarra Received: by thud.cs.utk.edu (5.61++/2.7c-UTK) id AA07974; Fri, 4 Dec 92 15:56:35 -0500 Date: Fri, 4 Dec 92 15:56:35 -0500 Message-Id: <9212042056.AA07974@thud.cs.utk.edu> To: mpi-comm@cs.utk.edu Subject: addresses for mpi I'm trying to put together a complete mailing list for MPI. Can you fill in the following information and send it to me. Thanks, Jack Name: Address: Email: Fax: Office phone number: Home phone number: (I'll use this only in an emergency and won't circulate.) From owner-mpi-comm@CS.UTK.EDU Mon Dec 7 21:39:33 1992 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA10542; Mon, 7 Dec 92 21:39:33 -0500 Received: by CS.UTK.EDU (5.61++/2.8s-UTK) id AA29754; Mon, 7 Dec 92 21:28:26 -0500 Received: from vnet.ibm.com by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA29750; Mon, 7 Dec 92 21:28:21 -0500 Message-Id: <9212080228.AA29750@CS.UTK.EDU> Received: from AUSVM6 by vnet.ibm.com (IBM VM SMTP V2R2) with BSMTP id 3637; Mon, 07 Dec 92 21:26:25 EST Date: Mon, 7 Dec 92 14:02:48 CST From: panda@vnet.ibm.com To: mpi-comm@cs.utk.edu To: mpi-comm@cs.utk.edu Subject: addresses for mpi I'm trying to put together a complete mailing list for MPI. Can you fill in the following information and send it to me. Thanks, Jack NAME: RAJ PANDA ADDRESS: E39/4305 IBM, 11400 BURNET RD., AUSTIN, TX 78758 EMAIL: PANDA@VNET.IBM.COM FAX: 1-512-823-6144 OFFICE PHONE NUMBER: 1-512-838-6632 HOME PHONE NUMBER: 1-512-835-1959 e.) From owner-mpi-comm@CS.UTK.EDU Mon Dec 7 22:10:17 1992 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA10659; Mon, 7 Dec 92 22:10:17 -0500 Received: by CS.UTK.EDU (5.61++/2.8s-UTK) id AA29855; Mon, 7 Dec 92 21:41:36 -0500 Received: from vnet.ibm.com by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA29851; Mon, 7 Dec 92 21:41:26 -0500 Message-Id: <9212080241.AA29851@CS.UTK.EDU> Received: from AUSVM6 by vnet.ibm.com (IBM VM SMTP V2R2) with BSMTP id 4638; Mon, 07 Dec 92 21:39:36 EST Date: Mon, 7 Dec 92 14:51:31 CST From: LBWARD@AUSVM6.VNET.IBM.COM To: mpi-comm@cs.utk.edu Subject: address for mpi mailing list Please add my name to the MPI forum. Linton Ward Name: Linton Ward Address: 11400 Burnet Rd, Austin TX 78758 Email: lbward@ausvm6.vmnet.ibm.com Fax: (512) 823-6144 Office phone number: (512) 838-5116 Home phone number: (512) 244-1949 From owner-mpi-comm@CS.UTK.EDU Thu Dec 10 13:10:06 1992 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA08984; Thu, 10 Dec 92 13:10:06 -0500 Received: by CS.UTK.EDU (5.61++/2.8s-UTK) id AA18058; Thu, 10 Dec 92 12:47:20 -0500 Received: from THUD.CS.UTK.EDU by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA18051; Thu, 10 Dec 92 12:47:17 -0500 From: Jack Dongarra Received: by thud.cs.utk.edu (5.61++/2.7c-UTK) id AA27178; Thu, 10 Dec 92 12:47:15 -0500 Date: Thu, 10 Dec 92 12:47:15 -0500 Message-Id: <9212101747.AA27178@thud.cs.utk.edu> To: mpi-comm@cs.utk.edu Subject: Information on the MPI meeting Here are some details on the MPI meeting which is set for January 6th-8th 1993 in Dallas. The meeting site will be the: Bristol Suites 7800 Alpha Road Dallas, TX 214-233-7600 The room rate is $89.00. When making a reservation tell them you are with the MPI meeting. TBS Shuttle Service will be providing complimentary shuttle service to and from the airports. If you fly into DFW, use their courtesy telephone and dial 03. If you fly into Love Field, you'll have to use a pay phone. They can be reached at 817-267-5150. Upon boarding the shuttle, refer to the MPI meeting. The registration fee for the meeting will be $75. Please make checks and POs payable to University of Tennessee. We will collect this at the meeting. The registration fee will go for coffee breaks, meeting rooms, AV and printer rentals. We should plan to start at 1:00 pm January 6th and finishing about noon on January 8th. The format of the meeting is: Wednesday, January 6 1:00 pm to 8:00 pm 2 Breakouts for 10 people 1 Breakout for 20 people 5:00 pm to 6:00 pm--On our own for dinner. Thursday, January 7 8:00 am to 10:00 pm 1 U-shape room for 40 people 6:00 pm to 8:00 pm-- The group dinner somewhere in the area. The hotel will provide round trip van transportation. Friday, January 8 8:00 am to 12:00 pm 1 U-shape room for 40 people This is an open meeting and all are welcome. For future planning here is a tentative list of dates, roughly 6 weeks apart, for the series of meetings: January 6-8 February 24-26 April 7-9 May 19-21 June 30-July2 If you have any questions, please feel free to contact me. Jack From owner-mpi-comm@CS.UTK.EDU Wed Dec 23 16:46:29 1992 Received: from convex.convex.com by mozart.convex.com (5.64/1.28) id AA20114; Wed, 23 Dec 92 16:46:25 -0600 Received: from CS.UTK.EDU by convex.convex.com (5.64/1.35) id AA05394; Wed, 23 Dec 92 16:46:20 -0600 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA14697; Wed, 23 Dec 92 17:31:12 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Wed, 23 Dec 1992 22:31:10 GMT Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from rios2.EPM.ORNL.GOV by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA14687; Wed, 23 Dec 92 17:31:08 -0500 Received: by rios2.epm.ornl.gov (AIX 3.2/UCB 5.64/4.03) id AA19206; Wed, 23 Dec 1992 17:29:47 -0500 Date: Wed, 23 Dec 1992 17:29:47 -0500 From: walker@rios2.epm.ornl.gov (David Walker) Message-Id: <9212232229.AA19206@rios2.epm.ornl.gov> To: abmacca@cs.sandia.gov, benson@rdvax.enet.dec.com, mpi-comm@cs.utk.edu, peter@sun.math.usfca.edu, srwheat@cs.sandia.gov Subject: MPI Information Status: RO Dear MPI committee member or observer, I need to know how many people to expect at the Dallas so please let me know if you intend to show up by sending email to walker@msr.epm.ornl.gov Please find below some information about the use of the MPI mailing lists, and who's involved in the various subcommittees. Please check that your entry is correct. If there are any errors please sent email to walker@msr.epm.ornl.gov and to dongarra@cs.utk.edu. Also there's some information about the January 6-8, 1993, meeting in Dallas. 1. Use of Mailing Lists ======================= The following mailing lists have been set up. mpi-comm@cs.utk.edu Whole committee (subcommitttes + observers) mpi-intro@cs.utk.edu Introduction subcommittee mpi-pt2pt@cs.utk.edu Point-to-point communication subcommittee mpi-collcomm@cs.utk.edu Collective communication subcommittee mpi-ptop@cs.utk.edu Process topology subcommittee mpi-lang@cs.utk.edu Language binding subcommittee mpi-formal@cs.utk.edu Formal language description subcommittee mpi-envir@cs.utk.edu Environment inquiry subcommittee All mail is being collected and can be retrieved by sending email to netlib@ornl.gov and in the mail message typing: send mpi-comm from mpi send mpi-intro from mpi ...etc. Also try: send index from mpi 2. Membership of Subcommittees (as of Dec. 22, 1993) ==================================================== 2.1. Introduction Subcommittee: Jack Dongarra (Chair) dongarra@cs.utk.edu Univ. of TN & ORNL Jim Cownie jim@meiko.co.uk Meiko Daniel Frye danielf@kgnvma.vnet.ibm.com IBM Kingston Ian Glendinning igl@ecs.soton.ac.uk Southampton Univ. Tony Hey ajgh@ecs.soton.ac.uk Southampton Univ. Robert L. Knighten knighten@ssd.intel.com Intel SSD Rusty Lusk lusk@mcs.anl.gob Argonne National Lab. Peter A. Rigsbee par@cray.com Cray Research, Inc. Steve Zenith zenith@kai.com Kuck & Associates, Inc. 2.2. Point-To-Point Communication Subcommittee Marc Snir (Chair) snir@watson.ibm.com IBM T. J. Watson Scott Berryman berryman@cs.yale.edu Yale Jaeyoung Choi choi@msr.epm.ornl.gov ORNL Jim Cownie jim@meiko.co.uk Meiko Jack Dongarra dongarra@cs.utk.edu Univ. of TN & ORNL Anne Elster elster@cs.cornell.edu Cornell Univ. Al Geist geist@msr.epm.ornl.gov ORNL Adam Greenberg moose@think.com Thinking Machines Corp. Bill Gropp gropp@mcs.anl.gov Argonne National Lab. Leslie Hart hart@fsl.noaa.gov NOAA/FSL Rolf Hempel hempel@gmd.de GMD Tom Henderson hender@fsl.noaa.gov NOAA/FSL Steven Huss-Lederman lederman@super.org Supercomput. Res. Ctr. John Kapenga john@cs.wmich.edu Western Michigan Univ. Robert L. Knighten knighten@ssd.intel.com Intel SSD Arthur B. Maccabe abmacca@cs.sandia.gov Sandia National Labs Oliver McBryan mcbryan@piper.cs.colorado.edu Univ. of Colorado Charles Mosher ccm@arco.com ARCO Paul R. Pierce prp@ssd.intel.com Intel SSD Peter Rigsbee par@cray.com Cray Research, Inc. Anthony Skjellum tony@cs.msstate.edu Mississippi State Univ. Vaidy Sunderam vss@mathcs.emory.edu Emory University Robert van de Geijn rvdg@cs.utexas.edu Univ. of Texas, Austin David Walker walker@msr.epm.ornl.gov ORNL Stephen R. Wheat srwheat@cs.sandia.gov Sandia National Labs Steve Zenith zenith@kai.com Kuck & Associates, Inc. 2.3. Collective Communication Subcommittee Al Geist (Chair) geist@msr.epm.ornl.gov ORNL Jaeyoung Choi choi@msr.epm.ornl.gov ORNL Jim Cownie jim@meiko.co.uk Meiko Mark Debbage md@ecs.soton.ac.uk Southampton Univ. Anne Elster elster@cs.cornell.edu Cornell Univ. Jon Flower jwf@parasoft.com Parasoft Corp. Adam Greenberg moose@think.com Thinking Machines Corp. Leslie Hart hart@fsl.noaa.gov NOAA/FSL C. T. Howard Ho ho@almaden.ibm.com IBM Almaden John Kapenga john@cs.wmich.edu Western Michigan Univ. Robert L. Knighten knighten@ssd.intel.com Intel SSD Arthur B. Maccabe abmacca@cs.sandia.gov Sandia National Labs Charles Mosher ccm@arco.com ARCO Steve Otto otto@cse.ogi.edu Oregon Graduate Inst. Paul R. Pierce prp@ssd.intel.com Intel SSD Peter Rigsbee par@cray.com Cray Research, Inc. Anthony Skjellum tony@cs.msstate.edu Mississippi State Univ. Marc Snir snir@watson.ibm.com IBM T. J. Watson Vaidy Sunderam vss@mathcs.emory.edu Emory University Robert van de Geijn rvdg@cs.utexas.edu Univ. of Texas, Austin David Walker walker@msr.epm.ornl.gov ORNL Stephen R. Wheat srwheat@cs.sandia.gov Sandia National Labs Steve Zenith zenith@kai.com Kuck & Associates, Inc. 2.4. Process Topology Subcommittee Rolf Hempel (Chair) hempel@gmd.de GMD Jim Cownie jim@meiko.co.uk Meiko Jon Flower jwf@parasoft.com Parasoft Corp. Tom Henderson hender@fsl.noaa.gov NOAA/FSL Steven Huss-Lederman lederman@super.org Supercomput. Res. Ctr. Robert L. Knighten knighten@ssd.intel.com Intel SSD Oliver McBryan mcbryan@piper.cs.colorado.edu Univ. Of Colorado Steve Otto otto@cse.ogi.edu Oregon Graduate Inst. Lew Tucker tucker@think.com Thinking Machines Corp. 2.5. Language Binding Subcommittee Scott Berryman (Chair) berryman@cs.yale.edu Yale Bruce Leisure bleisure@kai.com Kuck & Associates, Inc. Robert L. Knighten knighten@ssd.intel.com Intel SSD Cherri M. Pancake pancake@cs.orst.edu Oregon State Univ. Peter Rigsbee par@cray.com Cray Research, Inc. 2.6. Formal Language Description Subcommittee Steve Zenith (Chair) zenith@kai.com Kuck & Associates, Inc. Ian Glendinning igl@ecs.soton.ac.uk Southampton Univ. Tony Hey ajgh@cs.ston.ac.uk Southampton Univ. Robert L. Knighten knighten@ssd.intel.com Intel SSD Rusty Lusk lusk@mcs.anl.gov Argonne National Lab. 2.7. Environment Inquiry Subcommittee Bill Gropp (Chair) gropp@mcs.anl.gov Argonne National Lab. Daniel Frye danielf@kgnvma.vnet.ibm.com IBM Kingston Robert L. Knighten knighten@ssd.intel.com Intel SSD Steve Zenith zenith@kai.com Kuck & Associates, Inc. 2.8. Observers Ed Benson benson@rdvax.enet.dec.com DEC Clemens H. Cap cap@ifi.unizh.ch University of Zurich Michel J. Dayde dayde@enseeiht.fr ENSEEIHT-IRIT Jim Feeney feeneyj@gdlvm6.vnet.ibm.com IBM Kyle Gallivan gallivan@csrd.uiuc.edu CSRD, Univ. of Illinois Michael Heath heath@ncsa.uiuc.edu NCSA, Univ. of Illinois Mark Bowen Hill mbh@ecs.soton.ac.uk Southampton Univ. Shlomo Kipnis kipnis@watson.ibm.com IBM T. J. Watson ??? langley@dirac.scri.fsu.edu Florida State Univ. Bob Leary leary@sdsc.edu San Diego Super. Ctr. Lorie Liebrock lorie@cs.rice.edu Rice University Miron Livny miron@cs.wisc.edu Univ. of Wisconsin Ken Kennedy ken@rice.edu Rice University Neil MacDonald nbm@epcc.ed.ac.uk Univ. of Edinburgh Dan Cristian Marinescu dcm@cs.purdue.edu Purdue University Harish Nag harish@ssd.intel.com Intel SSD Dan Nessett nessett@llnl.gov LLNL Angela Ovearly fsang@kira.lerc.nasa.gov NASA Lewis Res. Ctr. Peter S. Pacheco peter@sun.math.usfca.edu Univ. of CA, San Franc. Raj Panda panda@vnet.ibm.com IBM Arnulfo Perez arperez@mtecv2.mty.itesm.mx Center for AI, Mexico Matthew Peters mpeters@vnet.ibm.com IBM UK Sci. Ctr. Jean-Laurent Philippe philippe@archipel.fr Archipel Robert S. Schreiber schreibr@riacs.edu RIACS, NASA Ames Mike Surridge ms@pac.soton.ac.uk Southampton Univ. Tricot tricot@archipel.fr Archipel Robert Voigt rvoigt@note.nsf.gov NSF Linton Ward lbward@ausvm6.vmnet.ibm.com IBM Dennis Weeks weeks@convex.com Convex Francis Wray falk@parsytec.de Parsytec 3. Institutions Involved in MPI =============================== 3.1. Universities Univ. of Edinburgh Emory University N.L. Mexico Centro de Intelligencia Artifical, Mexico Mississippi State University Oregon Graduate Institute Oregon State University Purdue University Rice University Western Michigan University Yale University University of Illinois, CSRD University of Illinois, NCSA University of Southampton University of Tennessee University of Texas at Austin University of Wisconsin University of Zurich University of Colorado 3.2. Laboratories ARCO Exploration and Production Technology Argonne National Laboratory Cerfacs ENSEEIHT-IRIT, France GMD Lawrence Livermore National Laboratory NASA RIACS NOAA Oak Ridge National Laboratory San Diego Supercomputer Center Sandia National Laboratories Supercomputer Research Center 3.3. Companies ARCHIPEL S.A. Cray Research, Inc. Digital Equipment Corporation IBM Austin IBM Almaden Research Center IBM Kingston IBM T.J. Watson Research Center IBM UK Scientific Centre Intel Supercomputer Systems Division Kuck and Associates, Inc. ParaSoft Corporation Parsytec Computer GmbH Thinking Machines Corporation 4. MPI Committee Meeting, Jan 6-8, 1993 ======================================= The next meeting of the MPI subcommittees will take place at Bristol Suites Hotel 7800 Alpha Road Dallas, Texas The meeting will start at 1pm on Wednesday January 6, 1993, and finish at noon on Friday, January 8, 1993. Rooms are $89 per night and reservations may be made by calling (214) 233-7600 (mention MPI meeting). The meeting registration fee will be $75. Please make checks and POs payable to University of Tennessee. Jack Dongarra will collect this at the meeting. The registration fee will go for coffee breaks, meeting rooms, AV and printer rentals. TBS Shuttle Service will be providing complimentary shuttle service to and from the airports. If you fly into DFW, use their courtesy telephone and dial 03. If you fly into Love Field, you'll have to use a pay phone. They can be reached at 817-267-5150. Upon boarding the shuttle refer to the MPI meeting. We have made arrangements with Delta Airlines to get a deal on airfares. When you book your flights please call Delta or have your travel agent call 1-800-241-6760 for reservations and refer to File Number S2145. The file goes by the name Message Passing Interface. A certain restrictions apply and seats are limited. An agenda will be sent out soon to all subcommittee members. We intend to rent a laser printer for use by attendees with protable computers. From owner-mpi-comm@CS.UTK.EDU Mon Jan 4 11:02:02 1993 Received: from convex.convex.com by mozart.convex.com (5.64/1.28) id AA06142; Mon, 4 Jan 93 11:02:00 -0600 Received: from CS.UTK.EDU by convex.convex.com (5.64/1.35) id AA08877; Mon, 4 Jan 93 11:01:56 -0600 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA15255; Mon, 4 Jan 93 11:54:20 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Mon, 04 Jan 1993 16:54:19 GMT Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from timbuk.cray.com by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA15247; Mon, 4 Jan 93 11:54:16 -0500 Received: from teak18.cray.com by cray.com (4.1/CRI-MX 2.4) id AA21055; Mon, 4 Jan 93 10:54:14 CST Received: by teak18.cray.com id AA05581; 4.1/CRI-5.6; Mon, 4 Jan 93 10:54:13 CST From: par@teak.cray.com (Peter Rigsbee) Message-Id: <9301041654.AA05581@teak18.cray.com> Subject: Profilers etc. (fwd) To: mpi-comm@cs.utk.edu Date: Mon, 4 Jan 93 10:54:10 CST X-Mailer: ELM [version 2.3 PL11b-CRI] Status: R James Cownie writes: > > I have an implementation issue which I would like to raise in the MPI > forum, since it is unclear where it fits into the sub-committee > structure I have mailed to both mpi-pt2pt and mpi-lang. Apologies to > those of you who receive this message twice. > > Issue > ===== > The major objective of MPI1 is to achieve portability of applications. > This has major benefits for us all (not least in legitimising and > therefore growing the MPP marketplace). > > One of the benefits which it would also be nice to achieve would be > the wide availability of different tools which support programming in > the MPI1 model. The most immediately obvious such tools (to me at > least !) are > > 1) HPF to Fortran + MPI1 translators > 2) Performance monitoring/tuning tools > 3) Debuggers > > Support for the first is easy (since it just requires what we're > already doing). > > Portable support for the second is not so trivial, since the > collection of useful performance information is much more intrusive. > > Portable support for the third is harder still, and I won't discuss it > further. First, I think this is an important issue. I'd argue that the need isn't necessarily to support portable tools, but it would be nice to define a mechanism by which performance and debugging tools could obtain at least the most basic information about the communications. Second, I don't think it really fits into either group that Jim sent the message to. The same issue, for example, also applies for global communications. This is why I copied the entire committee, and I think there should be some discussion in Dallas was to whether this is a topic that should be covered by the standard. Third, I fear this will be a very difficult subject to address. Clearly Jim had some specific use in mind when he made his suggestions, but didn't go into much detail (and properly so -- this isn't intended as a criticism). But there are some very basic differences between the ways that different tools are designed that need to be considered. What information should be recorded? Is data needed for each transmission (a "trace") or should it be accumulated by the library? Should the library pass information to the tool or should the tool query the library? I don't think there are clear cut answers here. But I think this is an important issue that should receive some airtime in Dallas. - Peter Rigsbee par@cray.com I've included the rest of Jim's proposal for those who didn't see the original mailing... > > Options > ======= > We have various possible options which we can take. > > 1) Ignore the problem > Provide no support for portable performance monitoring tools, and > leave each tool provider with a large porting problem. > > I don't like this solution, it loses some of the benefit of the > standard, which should be attracting people to build tools. > > 2) Document specific implementation hooks as part of MPI1. > In effect these would be callbacks from the library to profiling > code which could then do whatever it liked. > > 3) As 2, but without REQUIRING that a conforming implementation > provide the functions. They're there as a recommendation, rather > than being mandatory. > > > I think we should be concerned about this, and I'd like us at least to > make some recommendation. Personally I'd probably implement two > separate interface to the library, one of which provided the hooks, > and the other of which didn't so that you don't pay the cost of > checking the profiling hooks unless you asked to. > > Of course even if we do nothing it's not too hard to escape (using > horrible macros) in C, but Fortran doesn't always have macros, so a > properly specified internal solution is definitely preferable. > > Thoughts ??? > Flame me at Dallas. I'm travelling tomorrow. When are we having a > meeting in Europe ??? > > -- Jim > James Cownie > Meiko Limited > 650 Aztec West > Bristol BS12 4SD > England > > Phone : +44 454 616171 > FAX : +44 454 618188 > E-Mail: jim@meiko.co.uk or jim@meiko.com > > From owner-mpi-comm@CS.UTK.EDU Tue Jan 5 17:33:28 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA07561; Tue, 5 Jan 93 17:33:28 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA27130; Tue, 5 Jan 93 17:32:46 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Tue, 05 Jan 1993 22:32:45 GMT Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from fslg8.fsl.noaa.gov by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA27122; Tue, 5 Jan 93 17:32:42 -0500 Received: by fslg8.fsl.noaa.gov (5.57/Ultrix3.0-C) id AA23201; Tue, 5 Jan 93 22:32:36 GMT Received: by macaw.fsl.noaa.gov (4.1/SMI-4.1) id AA01387; Tue, 5 Jan 93 15:31:37 MST Date: Tue, 5 Jan 93 15:31:37 MST From: hender@macaw.fsl.noaa.gov (Tom Henderson) Message-Id: <9301052231.AA01387@macaw.fsl.noaa.gov> To: mpi-comm@cs.utk.edu Subject: A few general comments Here are a few comments/questions regarding the MPI1 draft proposal from folks here (Forecast Systems Lab). Apologies if you have seen them before! 1) Currently, communication between groups operating in different communication contexts appears to be very difficult. One way to alleviate this problem might be to provide a way to "name" contexts. Communicating groups could then refer to a context by name to ensure that the same message types were used. Message registration would also accomplish this and might be easier to implement (see our "Proposal 1" below). 2) We believe that MPI1 should include a specification of low level protocols that vendors should implement so that heterogeneous computing can be made available. Each media type (standard ethernet, FDDI, HIPPI, etc.) should have a specification. This would greatly aid in this standard being useful in a workstation environment. The protocol document would be a companion document to the syntactical standard for MPI1. 3) On p. 9 "No reference may be made to any process or group outside the current group context." How do different groups that are simultaneously working on different stages of a problem communicate? MPI_PARTG()?? 4) The use of stacks to manage contexts and groups may be a bad idea if MPI ever intends to support MIMD programs (how do groups synchronize stacks of message contexts?). Message registration would do the job without stacks. Registration of blocks of messages at the beginning of a program (or large module) would not affect performance very much. For groups, we like Al Geist's approach of identifying a process by one or more (group ID, process ID) pairs. We feel that communication between groups is very useful, even for SPMD programs. 5) It is clear that some way of responding to asynchronous events must be provided. We believe that "interrupt" type routines (such as Intel's hrecv() and hsend()) may not be the best approach. We have a brief alternative proposal for "Remote Services" below ("Proposal 2"). 6) The consensus seems to be that "parallel I/O" should not be a part of MPI1. Should it be a part of MPI2 or MPI3? If our intent is to create a standard that supports the development of portable code then we believe that I/O must be considered at some point. We should make sure that the specification of MPI1 includes the hooks necessary to include I/O at a later date. If our intent is to create only a low level library, then we can leave all this up to us users. We have some ideas about I/O and Data Decomposition below ("Proposal 3"). Successful implementation of such a proposal would require process topology information (yes, here's a plug for the Process Topology subcommittee). The proposals follow. Regards, Leslie Hart (hart@fsl.noaa.gov) Tom Henderson (hender@fsl.noaa.gov) Bernardo Rodriguez (bernardo@fsl.noaa.gov) PROPOSALS Proposal 1. Message Type Registration. Message types are managed using a global registry. Routines are provided that allow the user to "reserve" blocks of message types for use by the application. Message type registration is intended to prevent message type "collisions" when user code and third-party library routines are used together (historically, a serious problem with typed message passing systems). Initial concepts for message type registration are due to Tony Skjellum of Lawrence Livermore National Laboratory. Two proposed routines to register and monitor message types are described below. Arguments have been omitted for brevity. Message type registration routine: MPI_ReserveMsgType() Reserve one or more message types for use by the calling process. This routine is used to prevent message type "collisions" between user applications and libraries. Inputs include an ID string and the number of message types to be reserved. The routine first checks the locally maintained message type registry and returns an error condition if the ID string matches an existing entry. The routine then goes to the global message type registry to get the message types. If the ID string matches an existing entry in the global message type registry then the message types are copied from that entry to the calling process. Otherwise, new message types are registered with the ID string, entered into the global message type registry, and copied to the calling process. The new ID string and message types are entered into the locally maintained message type registry. Message type registration inquiry routine: MPI_InquireMsgTypeByName() Ask the message type registry to obtain the message types reserved under an ID string. An error is returned if no message types have been reserved under the ID string. Proposal 2. Remote Services (brief concept only). A remote service is similar in concept to a RPC (remote procedure call). Remote services are most often used to respond to specified asynchronous events. The actions taken in response to asynchronous events are performed by agent processes known as protocol handlers. A remote service is requested of one process by the same or another process. The remote service request is accompanied by a small amount of data to guide the action of the service. Routines are provided to allow user-created routine to be attached to specified requests. An attached user-created routine is called a protocol handler. A protocol handler does not have access to the address space of the its associated process. The protocol handler has the status of a full process (ie. it may perform I/O and message-passing communication). This is an advantage over some current interrupt-driven approaches. Remote service requests are handled one at a time on a first-come first-served basis (ie. a new remote service request will not be handled until all pending remote service requests are handled). This may be a drawback to this approach. Proposal 3. Some ideas about Cartesian Data Decomposition and Parallel I/O. Due to length, we have just included a few ideas. If there is any interest, we can provide (lots) more detail. 1) We propose Cartesian data decomposition routines to support the scalable distribution of multidimensional arrays among the processes in a group. These routines isolate the user from the details of implementing scalable data parallelism in a message passing environment. The routines also enhance performance by distributing data in a way that is most advantageous for a particular architecture. 2) A Cartesian data decomposition defines the distribution of a multidimensional data array onto a "logical" array of processes with the same dimension. The processes are members of a specified group. A Cartesian data decomposition is defined by information provided by the user and by run-time parameters determined by the system. The user specifies the group, array dimension, and number of elements in each dimension of the data array. The user may also optionally specify re-ordering of processes in the group and boundary behavior. The Cartesian data decomposition routines then combine this information with the number of processes in the group (provided by the system at run time) to determine which portion of the data array will reside on each process in the group. All information about the decomposition is then stored in a structure (or FORTRAN77 array). The "logical" array of processors is created using the Processor Topology routines (this idea has already been discussed in some detail in that subcommittee). 3) Cartesian data decomposition and I/O should be intimately related. Once a decomposition structure has been created, I/O should be possible using routines with syntax similar to this: cget(file, array, decomp) cput(file, array, decomp) where decomp is the decomposition structure. 4) Data decomposition tools have been available for several years, most notably from Parasoft's Express software package. These tools are extremely useful for almost all data parallel applications. Use of these tools in conjunction with I/O greatly simplifies program development on message passing systems. 5) Few areas in parallel processing have been dealt with in less detail by standardization efforts than parallel I/O. We would like to see behavior defined for such operations as seek/read and seek/write when several processes are accessing the same file. The concept of "loose synchronization" (Express and elsewhere) has proven to be very useful and should be included in a standard. Encapsulation of I/O in standardized routines would also permit performance optimization by the people who (should) know best how to do it (ie. the hardware vendors). From owner-mpi-comm@CS.UTK.EDU Wed Jan 6 11:42:45 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA24005; Wed, 6 Jan 93 11:42:45 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA12383; Wed, 6 Jan 93 11:41:35 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Wed, 06 Jan 1993 16:41:34 GMT Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from pnlg.pnl.gov by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA12375; Wed, 6 Jan 93 11:41:32 -0500 Received: from carbon.pnl.gov (130.20.65.121) by pnlg.pnl.gov; Wed, 6 Jan 93 08:39 PST Received: from fermi.pnl.gov by carbon.pnl.gov (4.1/SMI-4.1) id AA25662; Wed, 6 Jan 93 08:38:48 PST Received: by fermi.pnl.gov (4.1/SMI-4.1) id AA18870; Wed, 6 Jan 93 08:38:46 PST Date: Wed, 06 Jan 93 08:38:45 -0800 From: Robert J Harrison Subject: Re: Comments on Bill Gropp's comments To: mpi-comm@cs.utk.edu Message-Id: <9301061638.AA18870@fermi.pnl.gov> X-Envelope-To: mpi-comm@cs.utk.edu At the end of message <9301051955.AA01014@SSD.intel.com> Paul Pierce wrote > > I strongly recommend that we avoid separated info calls. Info can be returned > through output parameters, although that does annoyingly complicate receive > calls. Alternatively, info can be associated with handles, but that seriously > complicates handle management. (Handle management is a good topic for a > separate discussion.) I also strongly favour not having separate info calls. The necessary information is brief and does not excessively clutter argument lists, and can always be hidden in a wrapper if it is routinely not required. Also, by removing sources of ambiguity we give more freedom for implementation specific optimizations. > > If we decide we must have separated info calls, we should use dynamic context > semantics. > Yes to this too. Robert Harrison. From owner-mpi-comm@CS.UTK.EDU Wed Jan 6 17:17:30 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA10712; Wed, 6 Jan 93 17:17:30 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA29683; Wed, 6 Jan 93 17:16:48 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Wed, 06 Jan 1993 22:16:46 GMT Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from p.lanl.gov by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA29672; Wed, 6 Jan 93 17:16:45 -0500 Received: from beta.lanl.gov by p.lanl.gov (5.65/1.14) id AA11282; Wed, 6 Jan 93 15:16:41 -0700 Received: by beta.lanl.gov (5.57/Ultrix2.4-C) id AA29872; Wed, 6 Jan 93 15:15:46 -0700 Date: Wed, 6 Jan 93 15:15:46 -0700 From: sp@beta.lanl.gov (Stephen W Poole) Message-Id: <9301062215.AA29872@beta.lanl.gov> To: mpi-comm@cs.utk.edu I would like to have my name added to the mailing list. Steve... From owner-mpi-comm@CS.UTK.EDU Fri Jan 15 15:31:08 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA17910; Fri, 15 Jan 93 15:31:08 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA13121; Fri, 15 Jan 93 15:29:52 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Fri, 15 Jan 1993 15:29:51 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from rios2.EPM.ORNL.GOV by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA13112; Fri, 15 Jan 93 15:29:50 -0500 Received: by rios2.epm.ornl.gov (AIX 3.2/UCB 5.64/4.03) id AA13694; Fri, 15 Jan 1993 15:29:49 -0500 Date: Fri, 15 Jan 1993 15:29:49 -0500 From: walker@rios2.epm.ornl.gov (David Walker) Message-Id: <9301152029.AA13694@rios2.epm.ornl.gov> To: mpi-comm@cs.utk.edu Subject: Communication contexts I thought the discussion of communication contexts at the last Dallas meeting was lacking in focus, and I'm afraid the existing subcommittees may not deal with this area. If there are enough people interested perhaps there should be a separate communication context subcommittee. If you have any strong opinions on this please let me know. Also let me know if you would like to be on the communication context subcommittee if one is established. On the topic of communication contexts I think (at least) 4 approaches have been proposed. 1) Implicit communication contexts as in MPI1 controlled by push/pop. 2) Explicit registration as in Zipcode 3) Explicit communication contexts should be merged with explicit group contexts. Since groups cannot communicate with each other a new communication context can be created by creating a new group with the same members as the current group. 4) Chuck Simmons of Oracle has suggested that communication contexts are really a particular type of thread, and so can be handled using existing threads packages. If you have any ideas, or are strongly for or against any of the above approaches let mpi-comm know so we can initiate a discussion of this topic. David Walker From owner-mpi-comm@CS.UTK.EDU Fri Jan 15 15:48:16 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA18263; Fri, 15 Jan 93 15:48:16 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA14054; Fri, 15 Jan 93 15:47:41 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Fri, 15 Jan 1993 15:47:39 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from antares.mcs.anl.gov by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA14046; Fri, 15 Jan 93 15:47:38 -0500 Received: from donner.mcs.anl.gov by antares.mcs.anl.gov (4.1/SMI-GAR) id AA03636; Fri, 15 Jan 93 14:47:35 CST Received: by donner.mcs.anl.gov (4.1/GCF-5.8) id AA00709; Fri, 15 Jan 93 14:47:32 CST Message-Id: <9301152047.AA00709@donner.mcs.anl.gov> To: mpi-comm@cs.utk.edu Subject: Communication contexts Cc: walker@rios2.epm.ornl.gov Date: Fri, 15 Jan 93 14:47:31 CST From: Rusty Lusk | | On the topic of communication contexts I think (at least) 4 approaches have | been proposed. | 1) Implicit communication contexts as in MPI1 controlled by push/pop. | 2) Explicit registration as in Zipcode | 3) Explicit communication contexts should be merged with explicit | group contexts. Since groups cannot communicate with each other | a new communication context can be created by creating a new group | with the same members as the current group. | 4) Chuck Simmons of Oracle has suggested that communication contexts | are really a particular type of thread, and so can be handled | using existing threads packages. | If you have any ideas, or are strongly for or against any of the above | approaches let mpi-comm know so we can initiate a discussion of this topic. I think that 1) contradicts our genral agreement to avoid state as much as possible. 4) is a problem on systems that don't support threads, and we have more-or-less agreed to be consistent with thread packages but not depend upon them. 3) should be discussed along with the converse: that contexts rather than groups should be the primary concept and groups defined in terms of contexts. David, perhaps you could create a mailing list on which this discussion can take place, parallel to the subcommittee mailing lists, so it doesn't have to go to the whole committee. Thanks. Rusty From owner-mpi-comm@CS.UTK.EDU Fri Jan 15 16:13:17 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA18561; Fri, 15 Jan 93 16:13:17 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA16784; Fri, 15 Jan 93 16:12:04 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Fri, 15 Jan 1993 16:12:03 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from convex.convex.com by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA16768; Fri, 15 Jan 93 16:12:01 -0500 Received: from mozart.convex.com by convex.convex.com (5.64/1.35) id AA21882; Fri, 15 Jan 93 15:11:54 -0600 Received: by mozart.convex.com (5.64/1.28) id AA01950; Fri, 15 Jan 93 15:11:17 -0600 Date: Fri, 15 Jan 93 15:11:17 -0600 From: weeks@mozart.convex.com (Dennis Weeks) Message-Id: <9301152111.AA01950@mozart.convex.com> To: mpi-comm@cs.utk.edu Subject: Communication contexts Cc: lusk@antares.mcs.anl.gov, walker@rios2.epm.ornl.gov Rusty Lusk writes: > David, perhaps you could create a mailing list on which this discussion can > take place, parallel to the subcommittee mailing lists, so it doesn't have to > go to the whole committee. Thanks. I believe the default inclusion of everyone is a good idea, although creating a separate mailing list might be appropriate if some mpi-comm members wish to "unsubscribe". In the meantime, I will add one brief opinion: I felt that the discussions of contexts in the topology subgroup were quite sufficient, and it would probably be better to have the single subgroup which covers both groups and contexts, so that both will appear in the standard in a style which does not make them conflict with each other, and/or to avoid a proposal where groups and contexts would be totally redundant. I definitely agree that it is an important topic, and that the discussions last week in Dallas were far away from getting a proposal for groups and contexts that would be unanimously acceptable, so much work remains to be done. From owner-mpi-comm@CS.UTK.EDU Fri Jan 15 17:01:57 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA19329; Fri, 15 Jan 93 17:01:57 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA21190; Fri, 15 Jan 93 17:00:57 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Fri, 15 Jan 1993 17:00:56 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from fslg8.fsl.noaa.gov by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA21182; Fri, 15 Jan 93 17:00:54 -0500 Received: by fslg8.fsl.noaa.gov (5.57/Ultrix3.0-C) id AA20492; Fri, 15 Jan 93 22:00:50 GMT Received: by macaw.fsl.noaa.gov (4.1/SMI-4.1) id AA14823; Fri, 15 Jan 93 14:59:49 MST Date: Fri, 15 Jan 93 14:59:49 MST From: hender@macaw.fsl.noaa.gov (Tom Henderson) Message-Id: <9301152159.AA14823@macaw.fsl.noaa.gov> To: mpi-comm@cs.utk.edu Subject: Re: Communication contexts > | > | On the topic of communication contexts I think (at least) 4 approaches have > | been proposed. > | 1) Implicit communication contexts as in MPI1 controlled by push/pop. > | 2) Explicit registration as in Zipcode > | 3) Explicit communication contexts should be merged with explicit > | group contexts. Since groups cannot communicate with each other > | a new communication context can be created by creating a new group > | with the same members as the current group. > | 4) Chuck Simmons of Oracle has suggested that communication contexts > | are really a particular type of thread, and so can be handled > | using existing threads packages. > | If you have any ideas, or are strongly for or against any of the above > | approaches let mpi-comm know so we can initiate a discussion of this topic. > > I think that 1) contradicts our genral agreement to avoid state as much as > possible. 4) is a problem on systems that don't support threads, and we have > more-or-less agreed to be consistent with thread packages but not depend upon > them. 3) should be discussed along with the converse: that contexts rather > than groups should be the primary concept and groups defined in terms of > contexts. > > David, perhaps you could create a mailing list on which this discussion can > take place, parallel to the subcommittee mailing lists, so it doesn't have to > go to the whole committee. Thanks. > > Rusty I'm not sure, but I believe that Zipcode uses push/pop to control contexts (Tony, please correct me if I'm wrong). I'd like to add a 5th approach, basically the one proposed by Paul Pierce: 5) Contexts are used exclusively to insure that message collisions will not occur if independently developed sub-programs are combined. Contexts and groups are orthogonal. Contexts and threads are orthogonal. Each message has an associated context and tag. Message context is managed by library routines and is completely out of a user's control. Message tag is selected by the user. The method of managing message contexts is a separate issue (assuming we want contexts). Existing proposals are: 5a) Stack-based management (objected to due to hidden states). 5b) Explicit registration with user-defined "names" (probably requires some communication). 5c) Explicit registration by a central authority ("dollar bill" registration mentioned by Jim Cownie.) There is enough confusion about contexts and groups that we may want to keep this discussion open to everyone. If the concensus is to make a separate mailing list, please both of us to it. Thanks, Tom Henderson hender@fsl.noaa.gov Leslie Hart hart@fsl.noaa.gov From owner-mpi-comm@CS.UTK.EDU Fri Jan 15 19:06:47 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA21669; Fri, 15 Jan 93 19:06:47 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA27214; Fri, 15 Jan 93 19:05:42 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Fri, 15 Jan 1993 19:05:41 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from Aurora.CS.MsState.Edu by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA27206; Fri, 15 Jan 93 19:05:40 -0500 Received: by Aurora.CS.MsState.Edu (4.1/6.0s-FWP); id AA00450; Fri, 15 Jan 93 18:05:36 CST Date: Fri, 15 Jan 93 18:05:36 CST From: Tony Skjellum Message-Id: <9301160005.AA00450@Aurora.CS.MsState.Edu> To: hender@macaw.fsl.noaa.gov, mpi-comm@cs.utk.edu Subject: Re: Communication contexts In Zipcode, contexts have a group-wide scope. Groups of processes can involve more than one context. For instance, if 10 processes are involved in several stages of a calculation, each stage might have a different notion of how to do messaging, but each would use all the processes. Groups are important, because they allow control over the scope of global operations. Contexts are important because they provide guarantees to sub-programs about restricted scope of messages. Contexts are added typing-like information, but which is controlled by the system, not ad hoc by each user program. When they determine to start messaging, sub-programs request a context from a "postmaster general" in Zipcode. Currently, this is a group-oriented request (loose synchronization) resulting in each group member getting the context, and other information. In a lower-level system, simpler tactics might be possible. The way we did it was supposed to avoid possibility of races or other problems like that. To build large-scale software, without globalization of the local use of messaging resources, and to provide scalable algorithms with efficient global operations, both groups and contexts are needed. The push/pop state is an implementation detail of little importance. -Tony From owner-mpi-comm@CS.UTK.EDU Sat Jan 16 02:07:09 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA26026; Sat, 16 Jan 93 02:07:09 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA11696; Sat, 16 Jan 93 02:06:16 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Sat, 16 Jan 1993 02:06:15 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from sol.cs.wmich.edu by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA11688; Sat, 16 Jan 93 02:06:13 -0500 Received: from id.wmich.edu (id.cs.wmich.edu) by cs.wmich.edu (4.1/SMI-4.1) id AA06036; Sat, 16 Jan 93 02:01:08 EST Date: Sat, 16 Jan 93 02:01:08 EST From: john@cs.wmich.edu (John Kapenga) Message-Id: <9301160701.AA06036@cs.wmich.edu> To: mpi-comm@cs.utk.edu hi; It was my feeling that the concepts of group and context, that were clearly understood differently by different people at the start of the discussions, became somewhat better agreed on as the pt2pt discussion took place. Groups - used to define groups of processors for collective communication primitives and to allow mapping topologies onto processors. This allows a group of processors to be used in isolation to compute together. Contexts - As Tom (and Tony) express below, > From: hender@macaw.fsl.noaa.gov (Tom Henderson) > Message-Id: <9301152159.AA14823@macaw.fsl.noaa.gov> > To: mpi-comm@cs.utk.edu > Subject: Re: Communication contexts > Status: R > > I'd like to add a 5th approach, basically the one proposed by Paul Pierce: > > 5) Contexts are used exclusively to insure that message collisions will not > occur if independently developed sub-programs are combined. Contexts and > groups are orthogonal. Contexts and threads are orthogonal. Each message > has an associated context and tag. Message context is managed by library > routines and is completely out of a user's control. Message tag is > selected by the user. The only expression stated so far about the difference between tags and contexts I had gleaned was that context should now be wildcardable (IE no -1 for All contexts) while a receive ALL or some other MASK variant would be allowed on tags. > > The method of managing message contexts is a separate issue (assuming we > want contexts). Existing proposals are: > > 5a) Stack-based management (objected to due to hidden states). > 5b) Explicit registration with user-defined "names" (probably requires > some communication). > 5c) Explicit registration by a central authority ("dollar bill" > registration mentioned by Jim Cownie.) Add: 5d) A context is not strictly managed, but use should follow suggested guidelines. This could allow static allocation of contexts when reasonable, but also support a context server (as the Postmaster General in Zipcode). Think of the static contexts or non-registrar provided contexts as reserved with respect to any registration system. This could be part of the MPI initialization call, if a registrar is included. The drawback here of course is the user could violate whatever guidelines were given. It seems to me most types of name registration or stack allocation schemes, that are not too restrictive and allows user code to interact, are likely to allow errant management of contexts as well. I expect that allowing 5d) would let strictly controlled context systems to be built over MPI. > From: Tony Skjellum > Message-Id: <9301160005.AA00450@Aurora.CS.MsState.Edu> > To: hender@macaw.fsl.noaa.gov, mpi-comm@cs.utk.edu > Subject: Re: Communication contexts > Status: R > > In Zipcode, contexts have a group-wide scope. Groups of processes can Of course if we eliminate groups from the pt2pt call then contexts may not be explicitly limited to groups in MPI. (though a context server could be asked to provide a context relative to a group). > involve more than one context. For instance, if 10 processes are involved > in several stages of a calculation, each stage might have a different notion > of how to do messaging, but each would use all the processes. Groups > are important, because they allow control over the scope of global operations. > Contexts are important because they provide guarantees to sub-programs > about restricted scope of messages. Contexts are added typing-like > information, but which is controlled by the system, not ad hoc by each > user program. When they determine to start messaging, sub-programs > request a context from a "postmaster general" in Zipcode. Currently, this ... > To build large-scale software, without globalization of the local use of > messaging resources, and to provide scalable algorithms with efficient > global operations, both groups and contexts are needed. The push/pop > state is an implementation detail of little importance. I expect the use push/pop for context state would make implementation in some environments hard and I have some concern about requiring registration on very large systems. I do think a strictly controlled stack context system could be built over 5d). The question about context management relates to how high a level MPI should be at. I see it at a low level, that higher level systems can be built on. That is one reason for keeping context management "advisory". > > -Tony > I liked the zipcode documentation. I think it would be a good idea to make it available on netlib. john From owner-mpi-comm@CS.UTK.EDU Sun Jan 17 11:46:39 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA04093; Sun, 17 Jan 93 11:46:39 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA09441; Sun, 17 Jan 93 11:43:57 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Sun, 17 Jan 1993 11:43:56 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from sol.cs.wmich.edu by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA09433; Sun, 17 Jan 93 11:43:55 -0500 Received: from id.wmich.edu (id.cs.wmich.edu) by cs.wmich.edu (4.1/SMI-4.1) id AA08023; Sun, 17 Jan 93 11:38:43 EST Date: Sun, 17 Jan 93 11:38:43 EST From: john@cs.wmich.edu (John Kapenga) Message-Id: <9301171638.AA08023@cs.wmich.edu> To: mpi-comm@cs.utk.edu Although the question of IO specification in the MPI-1 has been raised before, I think it still needs to be addressed. Omission of any IO requirements means no program can be portable. If IO specifications are not included the introduction should make a strong case for the reasons. Any attempt to specify high level parallel IO at this point in time seems premature. Perhaps there can be a middle ground. We could specify low level IO, compatible with ANSI/ISO/POSIX standards and perhaps allow an inquire function to test for its presence. I will suggest every process should have access to Level-1 IO, given in the following list, which works for c and c++. -------------------------------------------------------------------- Level-1 Input and Output A process should have access to: 1) open/close/flush read/write lseek 2) readv/writev 3) fcntl dup dup2 ioctl 4) fopen/fclose/fflush scanf/printf fscanf/fprintf sscanf/sprintf 5) varargs vscanf/vprintf vfscanf/vfprintf vsscanf/vsprintf Behavior of these functions is to match that in current standards (specific standard to be chosen). -------------------------------------------------------------------- This could be loosened to only require defined behavior when access is by a single process. A Like Level-1 IO can be specified for Fortran-77 as well. This type of facility is provided in many vendor environments and several portable environments as well. A prototype version of the Level-1 IO could be built using gnu sprintf/sscanf, MPI-1 message passing and the assumption that some process can do IO. -------------------------------------------------------------------- Level-2 Input and Output A case can be made for Level-2 Input and Output as well. This would be group IO, no claim would be made for efficiency, only that the behavior was defined and would not overload the communication network. All processes in a group would execute the same IO calls as in level-1 and the joint behavior would be defined. To avoid state a modified name for each function could be used. A generic level-2 IO could be built on level-1 and the rest of MPI-1. -------------------------------------------------------------------- I propose we include a Level-1 specification and I think we should consider Level-2 as well. I am willing to draft a Level-1 suggestion and a Level-2 suggestion. I'm not sure which mailing list to put this on, but I think the issue needs to be addressed quickly (even if the decision remains to ignore IO). john From owner-mpi-comm@CS.UTK.EDU Mon Jan 18 09:30:23 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA14975; Mon, 18 Jan 93 09:30:23 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA16251; Mon, 18 Jan 93 09:29:26 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Mon, 18 Jan 1993 09:29:25 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from rios2.EPM.ORNL.GOV by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA16243; Mon, 18 Jan 93 09:29:24 -0500 Received: by rios2.epm.ornl.gov (AIX 3.2/UCB 5.64/4.03) id AA12246; Mon, 18 Jan 1993 09:29:23 -0500 Date: Mon, 18 Jan 1993 09:29:23 -0500 From: walker@rios2.epm.ornl.gov (David Walker) Message-Id: <9301181429.AA12246@rios2.epm.ornl.gov> To: mpi-comm@cs.utk.edu Subject: MPI news 1) NEW SUBCOMMITTEE A new subcommittee on profiling has been set up. It is chaired by Jim Cownie of Meiko, and you can send email to it at mpi-profile@cs.utk.edu. Please let me know if you want to join 2) NEWCOMERS FILE If you are a newcomer to the MPI forum or want general information send the message send mpi-newcomers from mpi to netlib@ornl.gov 3) REVISED VERSION OF MPI1 TECH REPORT A revised version of the MPI1 paper by Dongarra, Hempel, Hey , and Walker is available from netlib. This is basically the original document with some unnecessary and inappropriate routines removed. The PostScript file can be obtained by sending the message send mpi1.ps from mpi to netlib@ornl.gov 4) NEXT MPI MEETING Here are some details on the MPI meeting which is set for February 17th-19th 1993 in Dallas. The meeting site will be the: Bristol Suites 7800 Alpha Road Dallas, TX 214-233-7600 The room rate is $89.00. When making a reservation tell them you are with the MPI meeting. TBS Shuttle Service will be providing complimentary shuttle service to and from the airports. If you fly into DFW, use their courtesy telephone and dial 03. If you fly into Love Field, you'll have to use a pay phone. They can be reached at 817-267-5150. Upon boarding the shuttle, refer to the MPI meeting. The registration fee for the meeting will be $75. Please make checks and POs payable to University of Tennessee. We will collect this at the meeting. The registration fee will go for coffee breaks, meeting rooms, AV and printer rentals. We should plan to start at 1:00 pm February 17th and finishing about noon on February 19th. The format of the meeting is: Wednesday, February 17 1:00 pm to 8:00 pm point to point subcommittee meeting 5:00 pm to 6:00 pm--On our own for dinner. after dinner: other subcommittees meet Thursday, Febrauary 18 8:00 am to 12:00 pm Collective communication subcommittee meeting 1:00 pm to 6:00 pm Subcommittee reports presented to the main group 6:00 pm to 8:00 pm-- The group dinner somewhere in the area. The hotel will provide round trip van transportation. Friday, Febraury 19 8:00 am to 12:00 pm Subcommittee reports presented to the main group For future planning here is a tentative list of dates, roughly 6 weeks apart, for the series of meetings: March 31-April 2 May 19-21 June 30-July2 From owner-mpi-comm@CS.UTK.EDU Mon Jan 18 14:20:28 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA21493; Mon, 18 Jan 93 14:20:28 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA27089; Mon, 18 Jan 93 14:19:01 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Mon, 18 Jan 1993 14:19:00 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from rios2.EPM.ORNL.GOV by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA27075; Mon, 18 Jan 93 14:18:59 -0500 Received: by rios2.epm.ornl.gov (AIX 3.2/UCB 5.64/4.03) id AA15953; Mon, 18 Jan 1993 14:18:58 -0500 Date: Mon, 18 Jan 1993 14:18:58 -0500 From: walker@rios2.epm.ornl.gov (David Walker) Message-Id: <9301181918.AA15953@rios2.epm.ornl.gov> To: mpi-comm@cs.utk.edu Subject: Contexts group A new subcommittee and associated email group has been set up to responsible for MPI communication contexts. The current group membership is as follows: tony@cs.msstate.edu dongarra@cs.utk.edu lusk@mcs.anl.gov weeks@convex.com hender@fsl.noaa.gov john@cs.wmich.edu csimmons@us.oracle.com walker@msr.epm.ornl.gov If you would to join (or leave) this subcommittee please send email to me. You can send email to the group at mpi-context@cs.utk.edu. You can get the archived email for this group by sending the following message: send mpi-context from mpi to netlib@ornl.gov Regards, David From owner-mpi-comm@CS.UTK.EDU Tue Jan 19 12:29:45 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA19476; Tue, 19 Jan 93 12:29:45 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA24138; Tue, 19 Jan 93 12:28:28 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Tue, 19 Jan 1993 12:28:27 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from fslg8.fsl.noaa.gov by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA24130; Tue, 19 Jan 93 12:28:25 -0500 Received: by fslg8.fsl.noaa.gov (5.57/Ultrix3.0-C) id AA26202; Tue, 19 Jan 93 17:28:13 GMT Received: by nipmuc.fsl.noaa.gov (4.1/SMI-4.1) id AA02004; Tue, 19 Jan 93 10:30:14 MST Date: Tue, 19 Jan 93 10:30:14 MST From: hart@nipmuc.fsl.noaa.gov (Leslie Hart) Message-Id: <9301191730.AA02004@nipmuc.fsl.noaa.gov> To: mpi-comm@cs.utk.edu, john@cs.wmich.edu > From: john@cs.wmich.edu (John Kapenga) > To: mpi-comm@cs.utk.edu > > Although the question of IO specification in the MPI-1 has been > raised before, I think it still needs to be addressed. Omission > of any IO requirements means no program can be portable. If IO > specifications are not included the introduction should make a > strong case for the reasons. We (Tom Henderson and myself) agree and have also suggested such a proposal. Ours had a little more high level stuff, but not much. We do think you need some specification of how basic I/O performs in a parallel environment to provide a high level of portability. We think opportunities for parallelism can be expressed as well by routines that are loosely snychronous. > ... > Any attempt to specify high level parallel IO at this point in time > seems premature. Perhaps there can be a middle ground. We could > specify low level IO, compatible with ANSI/ISO/POSIX standards > and perhaps allow an inquire function to test for its presence. > I am willing to draft a Level-1 suggestion and a Level-2 suggestion. > I think that is a good idea and would be interested in cooperating on such an endeavor. > I'm not sure which mailing list to put this on, but I think the > issue needs to be addressed quickly (even if the decision remains > to ignore IO). Neither am I, let's hope it's not another subcommittee ;-) > > john > Leslie Hart (hart@fsl.noaa.gov) From owner-mpi-comm@CS.UTK.EDU Tue Jan 19 12:32:14 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA19727; Tue, 19 Jan 93 12:32:14 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA24283; Tue, 19 Jan 93 12:31:27 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Tue, 19 Jan 1993 12:31:25 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from fslg8.fsl.noaa.gov by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA24275; Tue, 19 Jan 93 12:31:22 -0500 Received: by fslg8.fsl.noaa.gov (5.57/Ultrix3.0-C) id AA26204; Tue, 19 Jan 93 17:28:14 GMT Received: by nipmuc.fsl.noaa.gov (4.1/SMI-4.1) id AA02007; Tue, 19 Jan 93 10:30:15 MST Date: Tue, 19 Jan 93 10:30:15 MST From: hart@nipmuc.fsl.noaa.gov (Leslie Hart) Message-Id: <9301191730.AA02007@nipmuc.fsl.noaa.gov> To: mpi-comm@cs.utk.edu, bernardo@fsl.noaa.gov, hart@fsl.noaa.gov, hender@fsl.noaa.gov, csimmons@us.oracle.com Subject: Re: "Message Capsule" Proposal (long) > From: Charles Simmons > To: bernardo@fsl.noaa.gov, hart@fsl.noaa.gov, hender@fsl.noaa.gov > Subject: Re: "Message Capsule" Proposal (long) > > As feedback... > > The message capsule proposal might be a bit high level. E.g. it > seems to implement the topmost layer of the ISO 7-layer communications > model which deals with transmitting data between processes running > on heterogeneous machines. I'm not familiar with with the ISO 7-layer approach, but what we were proposing was a place for each local process (or thread) to store information about one or more messages. Other words for capsule might be "handle" or "envelop". > > " > 6.1 Advantages > > - The number of separate send and receive routines is greatly reduced > without sacrificing functionality. > - A user who is used to "common practice" can use the simplified > versions of the routines. > - A user who wants more flexibility only needs to learn about the features > required for his/her specific application. (For example, if I only need > contiguous messages, then I don't need to know anything about strided data > items. If the receiving process always knows the data description of > received messages, then I don't need to know about the probe and "info" > routines.) > - "Hidden states" are removed so multi-threaded applications won't get > confused. > - Encapsulation of features in message capsules allows new features to be > added later without modifying syntax of existing routines. A new feature > would require addition of one or more new routines to modify and examine > message capsules. > " > > While the number of separate send and receive routines is reduced, > the number of other routines are correspondingly increased (e.g. attach > and info routines). The increase in routines is not combinatorial, but rather additive. For three new modes each of which have two possible settings, there is a maximum of six (6) new routines (ok, maybe 12 if there are other inquiry functions). In another approach the number of routines would be *mulitplied* by two for each new mode (i.e. multiplied by 8). > > Even in the MPI approach, users can limit themselves to simple versions > of various rotuines. Agreed, but I don't think the document has been taken to heart as a syntactical or semantical starting point. > > I don't see how capsules keep multi-threaded applications from > getting confused. If you have two threads in a receiver that are > each waiting for messages that look pretty much the same to the OS > or compiler or library code, it would appear to be fairly easy to > give the messages to the wrong threads. The sender really needs > to attach a thread identifier to a message under certain circumstances. > This responds to some issues that hidden state cause with multithreading. Such as what does the last message mean? We don't claim that it deals with which thread should be a target in a send. It just removes confusion if you probe and then wish to receive the message that you probed about. > > [There was a brief thought I had that this proposal reminded me of... > For certain types of applications, it would be possible for the sender > to know the layout of data in a receiver's address space. In an RPC > application, the receiver could transmit a description of the layout > to the sender. In an SPMD application, the sender might just know without > being told. This would allow the sender to prepend the layout description > to her message, and would allow the receiving OS to splatter received data > directly into user address space.] > > G'luck, Chuck Thanks for your input, we wanted to start a discussion about these issues. (BTW, your response was directly to us, so we included it in its entirety to mpi-comm). Leslie Hart Tom Henderson Bernardo Rodriguez hart@fsl.noaa.gov hender@fsl.noaa.gov bernardo@fsl.noaa.gov From owner-mpi-comm@CS.UTK.EDU Tue Jan 26 16:04:52 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA21467; Tue, 26 Jan 93 16:04:52 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA20893; Tue, 26 Jan 93 16:03:52 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Tue, 26 Jan 1993 16:03:51 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from enet-gw.pa.dec.com by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA20885; Tue, 26 Jan 93 16:03:47 -0500 Received: by enet-gw.pa.dec.com; id AA27890; Tue, 26 Jan 93 13:03:06 -0800 Message-Id: <9301262103.AA27890@enet-gw.pa.dec.com> Received: from rdvax.enet; by decwrl.enet; Tue, 26 Jan 93 13:03:45 PST Date: Tue, 26 Jan 93 13:03:45 PST From: MPS ENGINEERING 223-4656 To: mpi-comm@cs.utk.edu Cc: benson@rdvax.enet.dec.com Apparently-To: mpi-comm@cs.utk.edu Subject: POSIX 1003.4 message queues considered ? Has anyone involved in the MPI effort looked into POSIX 1003.4 (Realtime) message queues ? -Ed From owner-mpi-comm@CS.UTK.EDU Wed Jan 27 12:03:40 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA08441; Wed, 27 Jan 93 12:03:40 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA09490; Wed, 27 Jan 93 12:02:33 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Wed, 27 Jan 1993 12:02:31 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from antares.mcs.anl.gov by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA09480; Wed, 27 Jan 93 12:02:24 -0500 Received: from donner.mcs.anl.gov by antares.mcs.anl.gov (4.1/SMI-GAR) id AA02128; Wed, 27 Jan 93 11:02:21 CST Received: by donner.mcs.anl.gov (4.1/GCF-5.8) id AA16675; Wed, 27 Jan 93 11:02:16 CST Message-Id: <9301271702.AA16675@donner.mcs.anl.gov> To: mpi-comm@cs.utk.edu Subject: minutes of January MPI meeting Date: Wed, 27 Jan 93 11:02:15 CST From: Rusty Lusk Minutes of the Message Passing Interface Standard Meeting Dallas, January 6-8, 1993 The MPI Standards Committee met in Dallas on January 6-8, 1993, at the Bristol Suites Hotel in North Dallas. This was the third meeting of the MPI committee, but the first following the format used by the High Performance Fortran Forum. There were both general meetings of the committee as a whole and meetings of several of the subcommittees. Because interest in the Point-to-Point communications and the Collective communications was so general, these met as committees of the whole. No formal decisions were taken at this meeting, but a number of straw votes were taken in the subcommittees. These are included as part of the reports on the work of the subcommittees. These minutes were taken by Rusty Lusk (lusk@mcs.anl.gov) with some additions by Bob Knighten. Marc Snir's notes on the point-to-point subcommittee meetings are included here as well. These minutes are quite long. If you want to see the important topics you can search for --- and this will quickly the lead to each topic (and a few other things.) January 6 --------- ------------------------------------------------------------------------------- General Meeting ------------------------------------------------------------------------------- The meeting was called to order by Jack Dongarra at 1:30. Jack Dongarra presented the rules and procedures that had been circulated in the mailing list. In general, they say that we intend to operate in very open fashion, following the example set by the High-Performance Fortran Committee. He also described the subcommittee structure. For details, see the mailing list, A tentative schedule for future meetings was presented, which was amended on the last day (see there). All meetings will be in Dallas at the Bristol Suites. Steve Otto will coordinate the production of the document. He will obtain a set of LaTeX macros from the HPF Committee and distribute them to the subcommittee heads. It was suggested by Bob Knighten that the Executive Director arrange for copies of all pertinent documents be provided at the meetings. Dennis Weeks, who is somewhat local (Convex), volunteered to help with the relevant copying. The attendees were: Ed Anderson Cray Research eca@cray.com James Cownie Meiko jim@meiko.co.uk Jack Dongarra UT/ORNL dongarra@cs.utk.edu Jim Feeney IBM-Endicott feeneyj@gdlvm6.vnet.ibm.com Jon Flower ParaSoft jwf@parasoft.com Daniel Frye IBM-Kingston danielf@kgnvma.vnet.ibm.com Al Geist ORNL gst@ornl.gov Ian Glendinning Univ. of Southampton igl@ecs.soton.ac.uk Adam Greenberg TMC moose@think.com Bill Gropp ANL gropp@mcs.anl.gov Robert Harrison PNL rj_harrison@pnl.gov Leslie Hart NOAA/FSL hart@fsl.noaa.gov Tom Haupt Syracuse U. haupt@npac.syr.edu Rolf Hempel GMD hempel@gmd.de Tom Henderson NOAA/FSL hender@fsl.noaa.gov C. T. Howard Ho IBM Almaden ho@almaden.ibm.com Steven Huss-Lederman SRC lederman@super.org John Kapenga Western Michigan Univ. john@cs.wmich.edu Bob Knighten Intel SSD knighten@ssd.intel.com Bob Leary SDSC leary@sdsc.edu Rik Littlefield PNL rj_littlefield@pnl.gov Rusty Lusk ANL lusk@mcs.anl.gov Barney Maccabe Sandia abmacca@cs.sandia.gov Phil McKinley Michigan State mckinlehy@cps.msu.edu Chuck Mosher ARCO ccm@arco.com Dan Nessett LLNL nessett@llnl.gov Steve Otto Oregon Graduate Institute otto@cse.ogi.edu Paul Pierce Intel prp@ssd.intel.com Peter Rigsbee Cray Research par@cray.com Ambuj Singh UC Santa Barbara ambuj@cs.ucsb.edu Marc Snir IBM snir@watson.ibm.com Robert G. Voigt NSF rvoigt@nsf.gov David Walker ORNL walker@msr.epm.ornl.gov Dennis Weeks Convex weeks@convex.com Stephen Wheat Sandia NL srwheat@cs.sandia.gov ------------------------------------------------------------------------------- Point-to-point subcommittee ------------------------------------------------------------------------------- Mark Snir called the meeting to order at 1:40 p.m. It adjourned at 4:10 p.m. It resumed the following morning at 9:10 a.m. and adjourned at 4:15 p.m. Marc Snir began by summarizing the decisions that we have to make: * which operations? send receive channels? sendreceive? info arguments operations on queues probe? * operation modes sync async local and/or global termination interrupt-driven? * message types (data types) structure of data in core buffer packing * send-receive matching type (We later decided to call this "tag".) sender? * correctness criteria (See Marc Snir's paper in handouts) * heterogeneous operations * name space how processes are addressed flat? structured? implicit/explicit * error handling * interaction with threads, interrupt handlers, remote signalling * special operations for high performance ready receiver? * process startup * syntax/style (The plan is to postpone this for this meeting.) We will prioritize this list and then go through them one by one. (The priorities assigned were more or less in the order listed above.) Two preliminary questions were then discussed: A. Must we worry about multithreaded environments? James Cownie pointed out that threads were coming, in almost all new systems. Most systems have threads now. It was proposed that a process, which could send and receive messages, should be an address space, so that individual threads would not be (MPI-) addressable. B. What about signals? Paul Pierce suggested that we discuss signals first: do we want to support send/receive from interrupt handlers? These two questions were then discussed at length. Dealing with threads argues against the notion of "last message", since that implies state is maintained by the system. There was general agreement that "state" was a ` bad thing, but arguments in favor of state are: Sometimes one doesn't want all of the information available after an operations, so it shouldn't be returned. Having lots of arguments to calls is bad, especially inout arguments. Ways to avoid state are: Structures could be returned Return individual arguments Return tag to do queries on (but they one needs to free it.) Additional out arguments (OK in Fortran 90, but not in C or f77) User passes in storage to be used (so he knows the address), and MPI provides inquiry functions [For more details, see Jim Cownie's mail message of January 4, 1993 entitled: Multifarious] There was a general agreement that system state decreases portability and manageability, and we should decrease it when we can. James Cownie said that We need a reentrant style, and Mark Snir suggested that we try to make all function calls reentrant. When queried no one in the group objected to trying to make all the functions that are introduced in MPI reentrant. Now we began going through the above-mentioned major topics. Which Operations? ----- ---------- We have send and receive. How about send-receive (also called shift)? It can be efficiently implemented, and buffer can be reused. There was a discussion of the "two-body" send-receive (exchange) and the "three-body" version (ring-shift). Variations include those in which the send-buffer is required to be the same as the received- buffer and those in which is is required to be disjoint from the receive-buffer. Al Geist: We should focus on *required* operations. Steve Otto replied that send-receive *is* a required operation. Using "exchange" can help avoid deadlock. It was agreed that there was no consensus on these issues and it was decided to defer this to the collective communication subcommittee. Operation Modes --------- ----- The next topic that Marc Snir raised for discussion was when do send and receive return. Marc described several options: For send: 1) return as soon as possible 2) return when send-buffer is clear 3) return when the corresponding receive has completed For receive: 1) return as soon as possible 2) return when the receive-buffer is full "Receive has completed" means "when the user knows". In other words, when the sender has returned from send, the receiver has returned from receive. There was a general discussion about whether 3) was necessary? dangerous? Robert Harrison said he believed that 3) was the minimal version that was truly portable. Steve Otto pointed out that 3) is CSP-like. Rusty Lusk said that 3) would be easier to prove things about than the others. Adam Greenberg and Paul Pierce pointed out that neither TMC nor Intel have implemented an operation depending on the behavior of the receiver. A straw vote was taken and the vote was 17-3 in favor of having 3) as an option. Marc Snir pointed out that in his original proposal send returns a handle and the status of the handle is then tested for completion of the send operation, and asked if this is this desirable. There was general agreement that something of this sort was desirable, but a variety of alternatives were mentioned It was pointed out that sometimes one wants to wait on multiple outstanding operations. Al Geist prefers separating "wait" into "sendwait" and receivewait" for code readability. Bill Gropp suggested that instead of using handles, one could supply a routine to be called when an operation completes. James Cownie: "This gets really hairy in Fortran". There was a discussion of probing multiple outstanding receives: If the receives return handles, h1 = recv( ... ) h2 = recv( ... ) wait ( h1 or h2) ? wait ( h1 and h2 ) is not needed. Jame Cownie proposed that we supply an operations to *wait* on a vector of handles, which would return one of those that have succeeded. It would return the handle, not the status. A straw vote as taken on this proposal, which passed 17 - 0. So we have: status (handle) wait (array of handles) The send specifies what completion of send means. Handles need to be freed. It was pointed out the only the existence of such an operation has been decided, the semantics are yet unspecified - e.g. issues such as fairness or what wait returns when several complete are not yet specified. There was a long discussion of cancellation of send and receive. It was observed that there are serious implementation problems because of race conditions, freeing resources, etc. A straw poll was taken on including cancel in the initial MPI. It failed 7-19. This was the end of the Wednesday afternoon point-to-point meeting. January 7 --------- The point-to-point subcommittee (now a Committee of the Whole) resumed at 9:15 a.m. on Thursday morning. Marc Snir opened the meeting and summarized the progress so far: 3 ways in which send can terminate sendreceive postponed no cancel of incomplete send operation status and wait (successful status accomplishes same as wait) We did not get to: channels (the idea of trying to bind as soon as possible as many parameters as possible, so that they can be reused.) probe readyreceive Marc noted that channels and readyrecv address similar issues. Probably want only one of these. Do we want either? Rolf Hempel observed that we don't need channels - can depend on operating system to cache the connection information when doing synchronous communication. Adam Greenberg replied: NO! Want to be able to do this all at user level without "smart" OS. Channel creation and use might look like: handle = send_init( ... ) start(handle) wait(handle) free(handle) This is an intermediate point between bundled send/receive and full named channels. Indeed there are many intermediate points based on various early bindings. Is there enough experience to justify a standard? Bob Knighten observed that there has been substantial experience with channels on the iWarp system. There was next a discussion of the ready-receiver semantics proposed by Lusk and Gropp in the handouts. Steve Huss-Lederman said that such operations could make a difference of as much as 25% for matrix multiplication on the Delta. Some doubt was expressed about the universality of this optimization. Question of use of readyrecv by naive users again. Cownie mentions experience again. Greenberg: facilities for efficiency should not make it difficult to write correct programs. Wheat: Don't penalize users who do understand and can take advantage of efficient procedures. General back and forth discussion. Two straw votes were taken: Ready-receiver operations passed 13 - 10 Channels passed 19 - 2 (Marc Snir will write up a detailed proposal) The next topic discussed was the probe operation. Do we want such an operation, and if so, what should be its semantics? Probing must "lock" the message that it finds, else the information returned by the probe may be unreliable. (Consider the multithreaded environment.) Bill Gropp pointed out that probe is often used to find out the length of a message so that a buffer of the appropriate size can be allocated. Marc Snir pointed out that this is a problem with the November document, that we need to know the length of a message ahead of time. Jon Flowers suggested the need for a blocking probe. What is needed is to probe and then to receive the message found via the probe: handle = probe(params) . . . recv(handle) release(handle) Marc Snir pointed out that the handle serves as a lock on the message. James Cownie pointed out that while we agreed to not have a cancel for a send, we do need to be able to cancel receives, since an outstanding receive is permission for the system to write in the user's address space, which is a permission the user may want to revoke. A straw vote was taken on the existence of some form of probe, and it passed 25 to 0. Send-Receive Matching ------------ -------- The next topic is the matching of send and receive. Currently we have to discuss matching on: tag sender group id context id We will also need to discuss the name space issue for messages. Here are three proposals for the predicate that determine whether a message matches a particular receive: 1) simple matching on fields 2) more general, with mask, etc. 3) user defined function Adam Greenberg said that at TMC: A user defined function is used by the system whenever a message is received by a node to decide if it is to actually be received by the application. The parameters to the user defined receive predicate are tag and group. Issue: If most information is encoded in the tag, then the tag protocol must be understood by all users involved in writing a particular application. True, but not a serious problem. Best to identify small class of specific matching parameters (e.g. group) and use tag for everything else. James Cownie pointed out that the matching function, if not too complicated, can (and is, on many systems) done by special communications processors. There was further discussion of the difficulties of having the system call user code for screening messages. Paul Pierce pointed out that receipt of a message by the hardware is a crucial point for performance. There was general discussion of alternative approaches to getting at least some of this. The question of need for this generality was also raised. TMC has a user who wants and uses his own predicate function. Possibilities: (a) select on mask for fields (including a don't care); (b) simple static logical operations on fields; (c) user defined (b) might be match = AND (( message(i) = pattern(i) ) OR mask(i)) fields A straw vote was taken on whether to pursue allowing user-defines predicates. It was decided 26-1 not to allow user-defined functions for this purpose. (b) deferred until a proposal is available. Marc Snir summarized that matching by tag is generally agreed on and that this is not the only item for selection. After some discussion, matching by sender was also generally agreed on. So now, how do we identify a sender? Rusty Lusk spoke in favor of a flat name space, so that processes could be addressed independently of group, etc. There ensued a general discussion of groups, contexts, and the name space. It was pointed out that the name space expected by send could be flat and groups could be implemented by a function that converted any structured name into a flat integer id. Other proposals were to to have name=(rank,gid) with the restriction that this name be usable only within the given group (gid) and the sender must be a member of this group. By default the group would be ALL. Other alternatives mentioned were name=(rank,ALL)=pid and name=(pid,context). This led to a general discussion of context and the relation to groups. Marc Snir pointed out that we could have pid pid,context in which context did not change the meaning of pid. Paul Pierce said that tags and contexts should be separated since they need to be handled in different ways. Marc Snir pointed out that there should be no "don't care" on context. There was a discussion of servers that can process "any" message. This also led into discussion of flat name space vs. hierarchical name space where we would have pid(group, rank) function. Can use context to define groups, but there are other uses as well. Why groups as well as context? What is the difference between context and groups? Cownie: Context is just another integer used in the same manner as tag. Not quite - it is reserved, but what is the meaning of "reserved"? Greenberg was concerned about connecting send/receive behavior with groups. Snir: Suppose a users wants to have two independently written subroutines that use the usual rank notation. Wheat: Similarly want to use rank notation when partitioning machine. Snir: Both contexts and groups are nice, but do we need both? Gropp: Problem with mixing two applications both of which use 0-based indexing will need a larger common name space when they need to communicate. There was a general discussion of the cost of contexts. Cownie observed that context is cheap if only used to distinguish code - obtain a unique context id for the code by means of the "one-dollar random number generator": each author obtains a one-dollar bill, copies the serial number, and then burns the bill. But in general context is not cheaper than groups. Someone asked about spawning additional processes while program is running? Various people raised the question: If use name=(pid,context), does context change the meaning of the pid (i.e. is pid context {or gid or ???} relative.) There was some discussion of message registration. Paul Pierce observed that tag vs. context is only matter of registration. He wants to divorce tag and context for safety. This implies that one cannot use wild card for selecting on context. Various people noted difficulties with mixing tag and context. Adam Greenberg offered: Proposal - always separate tag and context. Have a context, NONE, so that pid with context NONE is unmodified, but with other contexts the pid may be relative. [NONE, GLOBAL, BASE] tag, context - must match on context Several people noted that there are two very different uses of context - identification of distinct code and identification of a group of processors. There is state, even distributed state, associated with remapping of processors with groups. POSSIBLE FIELDS FOR SEND/RECEIVE: tag context id group - no wild card - set of processes - registration management - receive only from group - managed by system Marc Snir asked whether we could agree on what would be carried with a message: tag context (like tag, except no wild card; management to be determined) Two straw votes were taken: Having contexts passed unanimously. Having the context *not* modify the process id passed unanimously. Groups ------ Three alternatives: no groups (use send(pid(group, rank), ...) instead) group as explicit parameter in operations use contexts to implement groups The basic difference is: do we want to be able to select on group? Straw vote: yes: 10 no:11 on the capability of selecting by group. (Thursday lunch occurred here) Message Data Types ------- ---- ----- WHAT IS A BUFFER? (Language bindings are going to be important here.) There are many options to consider: a) contiguous bytes (non-controversial) General agreement that 0-length messages should be allowed. b) contiguous buffers of other (implementation specific?) units? b) stride? (parameters: base-address, element-length, stride, number-of-elements) c) basic data types? d) arbitrary structures? e) How will we specify data to be communicated in a heterogeneous environment? f) iovec structures (array of pointers and lengths, as in un*x iovec for reads and writes) Marc Snir pointed out that one possibility is to have separate pack/unpack routines and then just send contiguous buffers. Rusty Lusk pointed out that this requires a copy that may be unnecessary on certain machines. Two choices - pack scattered buffer and send OR send scattered buffer. If the second, then may need a pack that produces the descriptor of a scattered buffer to be used by the send scattered buffer. Straw poll: Use IOVEC type send. Passed 18-1 Basic data types were deferred. Marc Snir observed that up to this point, a message is a set of bytes in storage, but now we are about to consider more meanings: message = sequence of *values* Should we use the same calls for homogeneous and heterogeneous calls? Can we have a fast homogeneous implementation of the heterogeneous calls? Bill Gropp pointed out that the current testbed implementation does this. SEND vs. SENDCHAR, SENDREAL, . . . To be compliant with F77 need to have at least SENDCHAR for correctness (and this is a real issue, e.g. on VAX.) Strictly need to have different for each basic data type (but in practice this is not an issue.) But for other than CHARACTER there is also an efficiency issue. 1. F77 conformance 2. Special problem of CHARACTER 3. Performance 4. Heterogeneity (?) Postpone to language binding discussion. This led into the issue of the general problem of converting types between languages and machines! This in turn led to a discussion of XDR (and mention of other systems such as NDR, ...) XDR supports the basic types (INT, REAL, COMPLEX, CHAR, etc.), array constructors, pack/unpack routines, etc. Do we use the same calls for homogeneous and heterogeneous systems? Can we have a fast implementation of heterogeneous procedures for a homogeneous system? What about a "message envelope" that specifies the environmental aspects of messages (e.g. heterogeneity features such as XDR.) When we talk about heterogeneity, do we expect MPI libraries from different vendors on different machines to cooperate? Include general SEND as SENDBYTES? Agreed that do not want SEND in homogeneous to require type information needed for heterogeneous environment. There was a discussion of whether we have to pick an interchange format, for example XDR. There seemed to be some agreement that we do (as MPI implementations from different vendors have to be able to communicate with one another), but no vote was taken. Error Handling ----- -------- The main issue here is whether an error detected by an MPI routine should result in the calling of an error-handler or return of a return code. Other issues are how much of error handling should be standardized as opposed to implementation-dependent, and how much user control there should be over error-handling. There are two types of error environments - soft (recoverable) and hard (unrecoverable). In a soft error environment there is the opportunity for cleanup on the part of both the "application" and the system, while in the hard error environment the system will cleanup and terminate the application. Choices: An mpi routine always returns (though it may return with an error code.) An mpi routine may call an exception handler There may be a default exception handler and there could be a user-installable one as well. Library writers may want to handle errors differently from how a user program wants to handle them (or have them handled by the system). Robert Harrison described the error modes used in TCGMSG and p4: A process has a user-settable state that determines whether an error should result in a (hopefully) graceful takedown of the program or in a error return code. Paul Pierce described the Intel method which uses two syntactically distinct classes of functions. For one class an error results in a message being printed and the process in which the error occurred terminating. For the other class an error code is set. There was some discussion of the problem of maintaining state in a multithreaded environment. Two straw votes were taken: Do we want a version of MPI that calls an exception handler: yes: 23 no: 0 Do we want a version with return codes: yes: 19 no: 1 Specific discussion of modes or "shadow" routines was deferred. Correctness Criteria ----------- -------- This concerns defining what is a correct implementation of MPI An assumption that had to be restated several times during the meeting is that MPI assumes a reliable underlying communication system, i.e. MPI does NOT address what happens if that fails. Two specific topics are order of messages and resource bounds. There was discussion about whether order preservation is required; that is, for messages from one process to another, messages are received in the order they are sent. Maintaining message ordering is troublesome, but seems essential for conveniently writing reliable portable programs. But then comes the question of what exactly this means, particularly with multithreaded processes! What is the effect of probe on the ordering of messages? Straw vote in favor of requiring order preservation: yes: 23 no: 4 On the issue of correctness with regard to resource exhaustion, Marc Snir suggested the following example: Process 1 Process 2 --------- --------- send to 2 send to 1 recv recv What should an implementation be required to do with this program? On the CM-5 this will always deadlock. On Intel and Meiko machines this will "usually" work (but how does one specify exactly when it will work.) Exchange is an even nastier case. ------------------------------------------------------------------------------ Summary of both Wednesday and Thursday point-to-point subgroup meetings by Marc Snir 1. Multithreaded systems and signal handlers. Should these be of concern to us? No vote was taken, but the general feeling was that we should try to define the various communication calls so that they do not rule out the case where the communicating process is multithreaded. The implications seems to be that all calls should be made reentrant, and the communication subsystem is, from the view-point of the application code, stateless. (With one with one obvious exception, namely that due to posted receive or send buffers, and perhaps additional exceptions having to do with global "modes", like error handling mode. 2. Small or large library? No vote taken. The general feeling is that we should provide many options for the advanced programmer that wants to optimize code (otherwise, all "interesting" codes will use non-portable features, but set the syntax so that the use that uses only the "core" functions need not be burdened by the availability of advanced options. 3. What functions? Clearly, SEND and RECEIVE General sentiment that a combined send-receive would be nice ("most used function on CM"), but discussion postponed until we have a proposed definition: Do we we want an exchange (source=dest), or a 3-body function (source != dest), or allow for both? do we want send_buffer identical to receive_buffer, or disjoint from receive_buffer, or allow arbitrary overlap between the two? What attributes are shared by sent message and received message, if at all? WAIT, STATUS and PROBE functions, and persistent handles are discussed later. 4. What modes? We want blocking and nonblocking sends and receives (blocking -- returns when operation terminated; nonblocking -- returns as soon as possible and a different call is needed to terminate the operation). We want synchronous and asynchronous modes (Synchronous -- operation terminated when terminated at all participating nodes. Asynchronous -- operation terminated when terminated at the calling node; e.g. a send terminates asynchronously when the sender buffer can be reused. Please let me know if you dislike this terminology and prefer something like "local" and "global".) The vote went 17-2 toward having a synchronous SEND (completes when RECEIVE has completed, i.e. when the corresponding WAIT has returned, or STATUS has returned successfully.) We did not discuss whether we want all 4 combinations of blocking-nonblocking and synchronous-asynchronous, or just 3 (blocking synchronous, blocking asynchronous and nonblocking asynchronous). We did not discussed explicitly, but "kind of assumed" that any SEND mode can match any RECEIVE mode. 5. How does one complete a nonblocking operation? The SEND and RECEIVE nonblocking operations return a handle that can be used to query for completion. WAIT(handle) blocks until operation completed; STATUS(handle) returns as soon as possible, and returns an indication for successful completion. In addition, these operations return information on completed RECEIVES: tag, message length, etc. for the received message. The information is returned in a structure provided by the caller. After return of a WAIT or successful return of a STATUS the operation handle is freed; the system has no more information on the completed operation, and has freed all associated resources. A more complex WAIT is needed, that waits for the completion of one out of several pending operations. Proposed syntax is WAIT(array_of_handles) that returns information on which operation succeeded and its parameters (voted 17 to 0). No CANCEL operation -- Once a SEND or RECEIVE is posted, it must complete. (Voted 19 to 7. Some peoples asked to reconsider at least canceling posted RECEIVEs, even if posted SENDs must complete). 6. Additional operations "ready-receive" SEND. SEND with a promise that a matching RECEIVE is already posted (A program where such SEND occurs with no preceding matching RECEIVE is erroneous and, hopefully, the implementation detects this error.) The justification is "it exists on some machine" and "it can improve performance by 25% on Delta". Accepted by 13 against 10. Persistent handles. Created by SEND_INIT(params) (resp RECV_INIT(params). can now be repeatedly used to send/receive messages with these parameters, and then explicitly destroyed. Supported by 19 against 2. PROBE. Allows probing for messages available to receive. Justification - "provides a mechanism to allocate memory to a message of unknown length, before it is received". The proposed mechanism is PROBE(params) that returns a lock to a matching message if there is a matching message that can be received. This message is now locked and can only be received using this lock. This was voted 25 to 0. Some level of uncertainty whether we should also allow to unlock without receiving (why should one want to do this?) 7. What is the buffer argument in SENDs and RECEIVEs? A message is a sequence of values, and as a particular case which is of most interest for homogeneous systems, and for which the syntax ought be simpler, a message is a sequence of bytes. There are various ways of specifying this sequence of bytes. a. Contiguous message: Starting address and length b. Regular stride message: Starting address, number blocks, length of blocks, stride. Voted with no opposition. c. IOVEC: a list of entries, each of which describes a type a or type b message. Voted 18 against 1. There was no discussion on a concrete proposal for typed messages, short of agreement that there should be such. The standard is not going to propose a concrete encoding of typed messages, and a concrete mechanism for message exchange in heterogeneous systems. 8. Matching of SENDs and RECEIVEs. A SEND operation associates with a message the following attributes. a. Sender id. b. Tag c. Context The idea of associated a group id, too, was rejected 11 to 10. The RECEIVE criterion is a Boolean predicate on these attributes of the form. (SENDER_ID = param1) and (TAG = param2) and (CONTEXT = param3). Don't cares are allowed for sender_id and tag, but not for context. Sender_id is determined by system, in the obvious manner, and is absolute (not relative to a group or a context). Tag is under sender control. Context is under sender control, but a yet to be determined mechanism is used to allocate valid context values to processes so as to prevent conflicts. All this was approved with no opposition. The idea of allowing the user to provide its own Boolean function as a receive predicate was rejected 26 to 1 (Reason: "hard to do if the matching is done by a communication coprocessor".) 9. Error handling a. We need a version of MPI where errors generate exceptions (user program halts when an error is detected in an MPI call, or a specific exception handling mechanism is invoked). Voted 19 to 1. b. we need to provide a version of MPI where calls return error codes, and do not cause exceptions, whenever possible. Voted 23 to 0. 10. Ordering of messages Messages sent from the same source to the same destination "arrive in the order they where sent". Voted 23 to 0. The exact implications in terms of order in which RECEIVEs can occur has to be worked out. It was pointed out that this condition may be somewhat hard to define in a multithreaded environment. End of Marc Snir's summary --------------------------------------------------------------------------- Collective Communication Subcommittee --------------------------------------------------------------------------- The Collective Communication Subcommittee was called to order by Al Geist at 4:30 p.m. on Wednesday. It continued until 6:40 p.m. when there was a break for dinner. The meeting resumed at 8:25 p.m. and finally adjourned at 10:10 p.m. Al Geist introduced this as the first meeting, since no real discussion on groups and collective communication took place in Minneapolis. One goal of this committee is to maintain consistency with the point-to-point operations. Any discussion of groups necessarily involves this subcommittee. Collective communication operations can be constructed out of the point-to-point primitives, but are desired because they can be implemented efficiently they are convenient for programmers. The committee then went through the set of collective communication primitives that had been proposed by Al Geist during the email discussions. Broadcast: info = MPI_BCAST(buf,bytes,tag,gid,root) On return, contents of buf for root is in buf for all processes. Al Geist pointed out that the group id here is explicit. Root has to be a member of the group. It was at this point that the committee decided that it would use the word "tag" for message type from now on to distinguish it from "type", which will now always mean type of data. Marc Snir pointed out that for consistency with point-to-point operations, there should be both local termination (the operation returns when the local process has done its part) and global termination (the operation returns when all processes have finished their participation) versions. There followed a discussion of the fact that the point-to-point committee seems to be adopting many different versions of send and receive, and that total compatibility will require many different versions of broadcast. There was a discussion of the reason for the tag parameter in the call. It is needed to disentangle multiple broadcasts occurring at approximately the same time. Paul Pierce described how the system can do this by generating sequence numbers. Others argued that the tag was useful for the programmer in any case, particularly for verifying program correctness. Marc Snir argued that there is a problem (because of the intuition that bcast provides barrier) 1 2 3 send(3) bcast rec(don't care) bcast send(3) bcast rec(don't care) Note that 3 may receive from 2 before 1, i.e. no barrier. Al Geist replied that we need barrier, but broadcast is NOT a barrier. James Cownie initiated a general discussion of whether broadcasts could be received by general receives. This would make it simpler to inherit some of the point-to-point semantics. Al Geist said that broadcast should be consistent with the other collective operations, all of which are symmetric. Paul Pierce suggested we specify collective communication routines in terms of model P-P implementation. This has consequences in terms of what options can be supported. Marc Snir pointed out that one can't actually specify collective communication in terms of point-to-point operations because they need dynamically-allocated additional space. It was decided to postpone a straw vote on whether all processes participating in a broadcast should do "broadcast" or only the root should "broadcast" and the others should "receive" because of concern about remaining issues, e.g. different varieties of recieves. The discussion of "error code" was deferred until the issue is settled in the Point-to-point communication subcommittee. MPI_GATHER: (see mail archives for details) It was proposed to have a version in which each participant contributes a different amount of information (a general "concatenate" function). Issues raised: How handle the situation where the number of bytes on each processor is different. How specify the type of data? For example one needs to know the size of the data type for various purposes, e.g. when doing recursive bisection. MPI_GLOBAL_OP: (see archives for definition) This does not include the data types. There was a discussion of how the forwarding processors know where to break buffers if the data type is not specified. Paul Pierce suggested that we should separate the case of user-defined combining operations from the system ones, which could be optimized. Robert Harrison suggested that the buffer be specified as (#items, length) at least for the user-defined operations. (Tag would be retained) Someone noted that "bytes" would be different on each processor in the heterogeneous case. Back to GATHER. Many agreed that the interface should be changed, but no proposal was offered. Straw vote on having separate general concatenation, to go along with the gather operation: yes: 18 no: 0 MPI_SYNCH There was general agreement that "BARRIER" would be a better name. James Cownie suggested that a tag argument would be helpful for debugging. There was also some discussion of failure of such a barrier, e.g. because some node fails. It was agreed that this was not a problem peculiar to this particular function. One individual nonetheless argued strongly for some kind of timeout for the barrier. Groups ------ gid = MPI_MKGRP (list of processes) There was much discussion of the format of the process list. As defined MKGRP defines a group as a subset of a pre-existing group. One alternative would be to allow creating a group consisting of processes from a number of other groups. (NB Identification of processes is unspecified. This is a task for the Point-to-point Communication Subcommittee.) MKGRP provides an implicit barrier among the processes joining the group. There are a number of problems about making sure that gid is uniform and known across the system. This is an efficiency issue. Should it be possible to SEND to a (gid,rank) pair? Marc argued that one should do Point-to-point communication only within a group, not between groups. Note that groups are constant - cannot add or delete members from a group. Also group creation is a barrier for the processes that are part of the group. This raises the question of how the processes joining the group know that they are joining. What is utility of groups? Certainly at present the only commonly used group is ALL. MPI_FREEGRP(gid) MPI_GRPSIZE MPI_MYRANK There was a general discussion of how group id's would be generated. Also a discussion of the mapping information: How to map back from my_rank and gid to rank in ALL? (In order to actually do a SEND.) ----- At this point the group broke for dinner ----- The continuation after dinner was an informal general discussion. There were some general question about experience from Al Geist to Paul Pierce. Adam Greenberg expressed interest in discussing channels. Channels are seen as an early binding, (Curryification) of various of the SEND/RECV functions which offer a number of gains in efficiency. There was a discussion of Fortran language bindings (F77, F90, HPF) of MPI. It was agreed by those knowledgable in the area that there are no special issues in regard to HPF. Steve Wheat discussed the Sandia implementation of channels on the Ncube. Sounds very similar to iWarp channels except that they are dynamic in creation. Jim Cownie noted that global-ops are going to result in non-determinism in numeric routines. Jim also elaborated on Meiko's BAD experience with ready_receive function - lots of user problems. Commonly user's try it on small problems and it works and speeds up. But then on large problems things erratically break and the user bitches. Paul Pierce noted that this is essentially Intel's force type and the Intel experience has not been so bad. In particular it is harder to use and does not generally work easily on small problems. Cownie: In general what to do when a ready_receive fails? No reasonable way to raise error. Response: Use a signal. Cownie: GAACK! This is implementation and not viable on all systems. John Kapenga listed six collective communication issues that he considers particularly important. [Missed the list] Other desirable collective communication features that were mentioned: global-exchange; all-to-all communication. What are criteria for including? Proposal: Difficulty of implementation; frequency of use; efficiency gain John Kapenga asked about 2-D and 3-D mesh operations - e.g. shifts? Adam Greenberg said this should be left to compilers. John: No Way! Adam argued that the compiler can recognize opportunity to avoid memory copies. Unless that same facility is available to user the compiler can do much better. The group adjourned at 10:10 p.m. --------------------------------------------------------------------------- Topologies Subcommittee --------------------------------------------------------------------------- The Topologies Subcommittee was called to order by Rolf Hempel at 4:00 on Wednesday. It lasted until dinner. --------------------------------------------------------------------------- Other Subcommittees --------------------------------------------------------------------------- The other subcommittees (Introduction, Formal Semantics, Environmental Enquiry, Language Binding) met informally after dinner on Wednesday. --------------------------------------------------------------------------- Meeting of the Whole Committee --------------------------------------------------------------------------- Thursday, January 7, 4:30 The Agenda for the rest of the meeting was presented: Introduction subgroup report Collective-communications subgroup report Process Topology subgroup report Environmental Inquiry subgroup report Formal Language subgroup report Language Binding subgroup report Profiling (Jim Cownie) Dates for future meetings Report of the Introduction Subcommittee: ------ -- --- ------------ ------------ Jack Dongarra presented the results of the subcommittee meeting that took place Wednesday night. This is essentially the draft that has been available from netlib for the last six weeks. There was some on-the-fly editing by the group at large. The goal of the Message Passing Interface simply stated is to develop a *de facto* standard for writing message-passing programs. As such the interface should establishing a practical, portable, efficient, and flexible standard for message passing. Goals ----- Design an application programming interface (not necessarily for compilers or a system implementation library). Allow efficient communication: Avoid memory to memory copying and allow overlap of computation and communication and offload to communication coprocessor, where available. Allow (but not mandate) extensions for use in heterogeneous environment. Allow convenient C, Fortran 77, Fortran 90, and C++ bindings for interface. Provide a reliable communication interface. User need not cope with communication failures. Such failures are dealt by the underlying communication subsystem. Define an interface that is not too different from current practice, such as PVM, Express, P4, etc. Define an interface that can be quickly implemented on many vendor's platforms, with no significant changes in the underlying communication and system software. The interface should not contain more functions than are really necessary. (Based on the latest count of send/receive variants, this drew a large laugh from the crowd.) Focus on a proposal that can be agreed upon in 6 months. Added: Semantics of the MPI should be programming language independent. Who Should Use This Standard? --- ------ --- ---- --------- This standard is intended for use by all those who want to write portable message-passing programs in Fortran 77 and/or C. This includes individual application programmers, developers of software designed to run on parallel machines, and creators of higher-level programming languages, environments, and tools. In order to be attractive to this wide audience, the standard must provide a simple, easy-to-use interface for the basic user while not semantically precluding the high-performance message-passing operations available on advanced machines. What Platforms Are Targets For Implementation? ---- --------- --- ------- --- --------------- The attractiveness of the message-passing paradigm at least partially stems from its wide portability. Programs expressed this way can run on distributed-memory multiprocessors, networks of workstations, and combinations of all of these. In addition, shared-memory implementations are possible. The paradigm will not be made obsolete by architectures combining the shared- and distributed-memory views, or by increases in network speeds. It thus should be both possible and useful to implement this standard on a great variety of machines, including those ``machines" consisting of collections of other machines, parallel or not, connected by a communication network. It was agreed that explicit remarks that MPI is intended to be usable with multithreaded processes and with MIMD (not just SPMD) programs should be added somewhere. What Is Included In The Standard? ---- -- -------- -- --- --------- The standard includes: Point-to-point communication in a variety of modes, including modes that allow fast communication and heterogeneous communication Collective operations Process groups Communication contexts A simple way to create processes for the SPMD model Bindings for both Fortran and C Support for Process Topologies In addition A model implementation and A formal specification. will be provided. It was proposed that explanation and rationale for the standard would also be provided as would sample programs and a validation suite. This is getting very ambitious. Jim Cownie also wants wrappers available for use by, for example, profiling. The suggestion is to provide "name shift", e.g. __MPI_SEND, etc. so the profiler can have MPI_SEND call __MPI_SEND after doing whatever is useful for profiling. What Is Not Included In The Standard? ---- -- --- -------- -- --- --------- The standard does not specify: Explicit shared-memory operations Operations that require more operating system support than is currently standard; for example, interrupt-driven receives, remote execution, or active messages Program construction tools Debugging facilities Tracing facilities Features that are not included can always be offered as extensions by specific implementations. Report of the Collective Communication Subcommittee: ------ -- --- ---------- ------------- ------------ Al Geist summarized the meeting that took place Wednesday afternoon (described above). Global functions beyond those discussed by the subcommittee, such as all2all or total_exchange, await written proposals. The (whole) committee added that Fortran 90 and HPF would be a good place to look for more combining functions (other than max, min, sum, etc.) It was agreed that a way to supply user-supplied functions would be useful. Issues mentioned include: What is a group? How are groups formed? Are group elements addressable, if so how? Are groups ordered (e.g. for prefix/suffix operations)? Group always an ordered subset of the ALL group? Partitioning? Connection with virtual topologies? This will be discussed when topology group reports. Friday, January 8 ------ ------- - Jack Dongarra called the meeting to order at 9:00. Report of the Process Topologies Subcommittee: ------ -- --- ------- ---------- ------------ Rolf Hempel reported on the meeting held Wednesday afternoon: Motivation: Applications have structures of processes Most natural way to address processes Processor topology is valuable to user Creation of subgroups is a natural way to implement topologies Draft proposal for MPI functions in support of process topologies (by Rolf Hempel) is in the handout bundle. The subcommittee made some changes to the draft. What functions should MPI contain? specification of logical process structure lookup functions for process id's clean interface to other parts of MPI (process groups) What should it not contain? any reference to particular hardware architectures algorithms for mapping of processes to processors If it does this, the user program will be portable, but will contain full information for processes mapping at the logical level. Claim: The use of process topologies is not an obstruction to quick implementation of MPI, since the first implementation can make random assignments. A process topology is assigned to a process group. Copying groups can be used to overlay different topologies on the same processes. All processes in a group call the topology definition function. Inquiry functions provide the translation of logical process location to process id. Supported Topologies: General graph structure: For each process, define the complete set of neighbors for each node. In principle this is sufficient as it covers all topologies. But it is not scalable as all processes have knowledge of all others. we should investigate a scalable version. However, important special cases should be treated explicitly, because regular structures can be specified in a scalable way easier to implement the mapping they cover a large number of applications. A special case: Cartesian structures grids/tori hypercube is a special case Support for creation of subgroups for regular structures will be useful. Special treatment for trees? deferred User-defined topology definition functions? deferred It will be necessary for the inquiry functions to provide information on the hardware topology, so that a user can provide his own mapping function. Marc Snir: We need to consider consistency of mapping alignments, for example an octtree for image processing with a grid structure. Al Geist: What is connection between group and topology. Recall that a group is a linear *ordered* array which is a kind of topology. General discussion of copying topologies and groups Proposal is to have at most one topology per group so can use group id as name for topology. This is reason that there must be a group copy. David Walker: We need closer coordination between the collective communication subcommittee and the topology subcommittee, since groups are central to both. Report of the Environmental Enquiry Subcommittee: ------ -- --- ------------- ------- ------------ Bill Gropp reported that the Environmental Enquiry subcommittee needs to wait and get a better picture of what MPI will contain. Jon Flower again asked for cpu_time. This was discussed, and we were reminded that these were more-or-less rejected at the Minneapolis meeting as not being part of MPI. Standardization should come from POSIX. Marc Snir: Part of the subcommittee's job should be to decide *what* can be enquired about as well as how it will be done. There was general discussion about inquiring about both MPI parameters and implementation parameters. Also if parameter *setting* as well as enquiry should be supported. (Buffer pool sizes, for example). Jon Flower also asked about system hints. He suggested it should be possible to tell the system about implementation specific tuning in a system independent way. Report of the Formal Specification Subcommittee: ------ -- --- ------ ------------- ------------ Rusty Lusk reported the committee was without its chairman, Steven Zenith, but that it viewed its mission as to try to formalize what the other subcommittees decide on. It will probably use CSP, for lack of experience with any other formal specification language. Bob Knighten suggested that the subcommittee look into LIS (Language Independent Specification) that POSIX defined in order to separate semantics from language bindings. Report on MPI -1 (minus one) ------ -- ------ ----------- James Cownie presented an MPI anti-specification. Ya hadda be there, but in case you weren't or just want to be reminded, here is a transcription of Jim's slides. MPI -1 (Jim Cownie) In the spirit of LPF (Low Performance Fortran) * Bindings ONLY for Mathematica Occam ML * No function take arguments or returns result * Point to Pointless communication * 1024 different sends NO receives * Full support for 0 dimensional topologies * User data in a message limited to 1 byte (of 6 data types) BUT 1 KByte of TAG, CONTEXT * Informal semantics - Formal Syntax * All groups are contexts * All contexts are groups * Non blocking wait * Non blocking barrier * All user programs are unsafe & erroneous, they therefore do all their work in the exception handler. --------------------------------------------------------------------------- A Profile/Instrumentation subgroup was formed with Jim Cownie as chairman. Steve Otto, as general editor, will contact subgroup chairmen to begin discussion of editing concerns. Discussion of meeting format. The following was proposed as a format for subsequent meetings, based on the experience with this meeting. Wed. afternoon: point-to-point Wed. night: all subcommittees other than pt-to-pt and collective comm. Thurs. morning: collective communication Thurs. afternoon: subcommittee reports Fri. afternoon: subcommittee reports Meeting Dates: It was decided to moved the next two meetings up a week from when they were tentatively scheduled. The next meeting will be Feb 17-19. The next one after that will be Mar 31-Apr 2 The currently-scheduled May 19-21 and June 30-July 2 meetings may also be moved up as well. Note that July 2 will be a holiday in the United States. From owner-mpi-comm@CS.UTK.EDU Wed Feb 3 01:42:10 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA07543; Wed, 3 Feb 93 01:42:10 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA03537; Wed, 3 Feb 93 01:41:08 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Wed, 3 Feb 1993 01:41:06 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from antares.mcs.anl.gov by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA03528; Wed, 3 Feb 93 01:41:05 -0500 Received: from donner.mcs.anl.gov by antares.mcs.anl.gov (4.1/SMI-GAR) id AA28816; Wed, 3 Feb 93 00:41:03 CST Received: by donner.mcs.anl.gov (4.1/GCF-5.8) id AA23933; Wed, 3 Feb 93 00:41:00 CST Message-Id: <9302030641.AA23933@donner.mcs.anl.gov> To: mpi-comm@cs.utk.edu Subject: A suggestion for a multi-level MPI Date: Wed, 03 Feb 93 00:40:59 CST From: Rusty Lusk At the last meeting, it became apparent that many people were uncomfortable with the large number of routines (1024 sends?) that we seemed to be going towards. There are good reasons for the standard not to be incomprehensibly complex. There are good reasons also for it not to be crippled by lack of functionality. Are we stuck? There are two ways to deal with this problem. This note is a proposal that we adopt both of them. The reason the problem can be solved is that the sets of options that we have been considering (blocking or not, contiguous buffer or not, heterogeneous machines or not, etc.) are orthogonal, and one of a large number of possible send routines can thus be specified simply and without extra parameters. In addition, certain subsets of the set of all operations can be identified and packaged for ease of use. Here is a suggestion for how to organize the routines and end up with a powerful, flexible, yet easy-to-use library of point-to-point functions. (We assume that these will all be renamed to have an MPI prefix, to avoid name-space pollution) Level 4 (very restrictive, very simple): send(dest,tag,buf,len) recv(dest,tag,buf,len) Both of these block until the operation is locally complete. The buffer is a contiguous sequence of bytes. No special protocols or translation for heterogeneous machines. This level is enough to express lots of parallel algorithms. The main reason we need more is for efficiency and control. We now add these in steps. Level 3 (still simple, but allows for overlap of computation and communication and better buffer management, and allows heterogeneous machines to communicate.) bsend(tag,dest,bufadd,buflen) nsend(tag,dest,bufadd,buflen) bsendh(tag,dest,bufadd,datatype,numitems) nsendh(tag,dest,bufadd,datatype,numitems) brecv(tag,dest,bufadd,buflen) nrecv(tag,dest,bufadd,buflen) brecvh(tag,dest,bufadd,datatype,numitems) nrecvh(tag,dest,bufadd,datatype,numitems) Here the n or b selects block or nonblocking operations, h specifies translation for heterogeneous machines. The restrictions (which allow the parameter lists to be short and the number of routines to be small) are: no noncontiguous buffers, data in message all the same type, no globally synchronized, CSP-like send-receive, and no special protocol (like the ready-receiver protocol discussed at the last meeting). In order to emphasize the orthogonality, we might organize these routine names in a diagram like this: [n][send][ ] [b][recv][h] That is, one makes a choice in each column. Level 2 (access to almost all MPI functionality via a large number of orthogonally named routines): [c][n][send][ ][ ] [s][b][recv][h][rr] [g][s] the first column designates the buffer type (contiguous, strided, general gather/scatter), the second is the termination type (nonblocking, blocking, synchronized), then send or receive, "h" or not for heterogeneous communication, "rr" or not for "ready-receiver" protocol. This is a lot of routines (3 x 3 x 2 x 2 x 2 = 72) but because of orthogonality it is easy to understand. The only restriction here is that there are no "channels": the initialization of a send operation is combined with initiating it. Level 1 (full MPI functionality, in a small, flexible set of routines) handle = init_send(tag,dest) mod_send(handle,option,value) do_send(handle) free_send(handle) along with the corresponding receive routines. The idea is that the "init" routines will set up a usable set of defaults for all the options, specifying only minimal parameters, and then the "mod" routine can be repeatedly called to specify all the options with regard to buffer type, termination type, etc. The point is that at this level we can offer not only all possible options, with a small number of routines, and include the "setup" routines proposed by Marc for increased efficiency, but we also allow room for growth in the standard through the addition of more values for "option" in the "mod" calls. So here's how it might look, this time from the bottom up: Level 1: full functionality, small number of routines, channel setup Level 2: sequences of Level 1 calls of the form init... mod... mod... mod... do... are given specific names in an organized way. This promotes ease of use and also efficiency (fewer calls). Level 3: several of the Level 2 routines are renamed and promoted to this level to provide a useful subset of MPI functionality in a small number of routines, each having the minimal number of parameters. Level 4: ultra-simple interface All four levels would be part of the standard and it would be possible to mix levels in the same program. We have not considered carefully yet how this interacts with groups and contexts, but similar ideas might be useful there, where there are similar problems in providing both functionality and simplicity. Comments? Rusty Lusk and Bill Gropp From owner-mpi-comm@CS.UTK.EDU Wed Feb 3 06:16:01 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA27255; Wed, 3 Feb 93 06:16:01 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA23277; Wed, 3 Feb 93 06:14:47 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Wed, 3 Feb 1993 06:14:46 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from gatekeeper.oracle.com by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA23269; Wed, 3 Feb 93 06:14:40 -0500 Received: from wrpyr.us.oracle.com by gatekeeper.oracle.com (5.59.11/37.7) id AA21815; Wed, 3 Feb 93 03:14:37 PST Received: by wrpyr.us.oracle.com (5.59.11/37.7) id AA15597; Wed, 3 Feb 93 03:17:14 PST Message-Id: <9302031117.AA15597@wrpyr.us.oracle.com> Date: Wed, 3 Feb 93 03:17:14 PST From: "Chuck Simmons " To: mpi-comm@cs.utk.edu Subject: Re: A suggestion for a multi-level MPI Reply-To: csimmons@us.oracle.com Original-To: WRPYR:mpi-comm@cs.utk.edu In-Reply-To: WRPYR:owner-mpi-comm@CS.UTK.EDU's message of 02-03-93 00:40 Rusty, Bill -- I do view the interface as being layered, but I would organize the layers differently. And I have a couple of arguments that I haven't tried out yet against some of the send/recv routines. First, it sounds to me like you are suggesting that the 72 routines be implemented as 72 simple routines that each make a small number of subroutine calls to very simple routines. And then the do_send routine would do a lot. I guess the do_send routine on a typical machine might look like: /* convert the data into a vector of buffers */ if (options & MPI_OPT_ONEBUF) { iovlen = 1; iovp = bufp; } else if (options & MPI_OPT_STRIDED) { /* malloc local buffer, gather data, set up iovlen and iovp */ } /* translate the data */ if (options & MPI_OPT_HETER) { /* malloc new buffer, translate data into new buffer */ } /* send the data non-blocking and unreliable */ if (options & MPI_OPT_RR) { sendrr (...); } else send (...); /* block if so requested */ if (options & MPI_OPT_BLOCK) { block (...); } My view of the layering is: basic functionality: non-blocking vector send non-blocking vector receive easy-to-use basic functionality: non-blocking single buffer send non-blocking single buffer receive reliable: synchronous vector send synchronous vector receive easy and reliable: synchronous single buffer send synchronous single buffer receive In the above, each of the later listed routines can be implemented on top of an earlier listed routine. I would then examine each of the remaining aspects of the functionality: heterogeneity, ready-receiver, strided messages, and blocking non-synchronous messages. For heterogeneity, I would argue that it is wrong to put the datatype translation filter in the communications channel. In your model, it is relatively easy to do things like: bsendh (dest, tag, buf, FLOAT, 1); or even: bsendh (dest, tag, buf, ARRAY|FLOAT, nitems); but what about bsendh (dest, tag, buf, rpc_t, 1); where 'rpc_t' is some user defined data structure? Thus, this model lacks generality. Only certain simple messages can be heterogeneously sent efficiently through the interface. A completely different paradigm must be used if you want to send a structure instead of an array. Further, consider how the "bsendh (dest, tag, buf, ARRAY|FLOAT, nitems);" routine is going to be implemented on a typical machine: buflen = sizeof (net_float) * nitems; sendbuf = (net_float *) malloc (buflen); if (!sendbuf) return ENOMEM; for (i = 0; i < nitems; i++) { sendbuf[i] = float2netfloat (buf[i]); } bsend (dest, tag, (void *)sendbuf, buflen); free (sendbuf); (In the above, we assume that a "net_float" holds the network normalized format of a floating point number.) It would be as efficient to implement the translation routines completely separate from the communications routines. But, wait, you say, our machine is not typical and we can implement this as: bsend (dest, tag, &header, sizeof (header)); for (i = 0; i < nitems; i++) { bsend (dest, tag, float2netfloat (buf[i]), sizeof (net_float)); } where 'bsend' is assumed to be a machine instruction. My argument against this is that the gain in efficiency is sufficiently small that it's not clear its worth the added complexity. In both implementations, the cpu time will be dominated by the data conversion. In the "typical" implementation, there is an additional cost for moving data between two buffers, but that data movement occurs at the speed of local memory access which is likely to be much higher than network bandwidth. Thus, it is much easier for the hardware to DMA the first implementation and run some other process while that DMA is in progress. In the second implementation, the sending process is essentially hogging the cpu for the full time it takes to transmit each piece of the message. Obviously, if there exists a standard for network normal formats of datatypes, and if hardware vendors implement the data translation facilities in hardware, then we could DMA out the whole message heterogeneously. But surely there are other things that we would rather have hardware vendors implementing for us. The above is essentially the same argument that I'm going to use for strided messages. Strided messages are only efficient if you have hardware that can DMA the strided vector, and then its not so efficient that it's obviously worth the effort. For ready-receiver, I'm going to argue that the communications subsystem should always assume the receiver is ready and then handle the case where this isn't true. If the application has set things up so that this is true, that application will run efficiently whether or not the sender specifies that the receiver is ready. If the application isn't being careful, it will run slowly in your model, but in my model, it will usually run quickly since memory is usually availabe in the receiver, and my model will run as slowly as your model when congestion is present at the receiver. Implementing the ready-receiver places strong semantics on the sender and receiver. The sender has to know whether or not the receiver was implemented in a fashion that allows the sender to assume that the receiver is ready. Finally, let's consider synchronous messages versus blocking messages. For most applications, you want reliability. Implementing blocking non-synchronous messages doesn't help you implement this. You can't reuse the buffer being sent until you know that it has been received at the other side. So you might as well either send the message synchronously, or send it non-blocking and go off and do something else while you're waiting to see if you'll need to retransmit the buffer. [Hmmm... I did mention, didn't I, that I was still trying out a couple of these arguments...] This reduces the number of interface routines to 8. I could probably be convinced that adding 4 more interface routines to support strided messages doesn't overly complicate the interface. -- Chuck *** We use Oracle*Mail running on Oracle V6.2 and an nCUBE2 supercomputer. *** ---- Included Message ---- Received: 02-02-93 22:48 Sent: 02-03-93 00:40 From: WRPYR:owner-mpi-comm@CS.UTK.EDU To: mpi-comm@cs.utk.edu Subject: A suggestion for a multi-level MPI Reply-To: WRPYR:owner-mpi-comm@CS.UTK.EDU X-Resent-To: mpi-comm@CS.UTK.EDU ; Wed, 3 Feb 1993 01:41:06 EST Errors-To: owner-mpi-comm@CS.UTK.EDU At the last meeting, it became apparent that many people were uncomfortable with the large number of routines (1024 sends?) that we seemed to be going towards. There are good reasons for the standard not to be incomprehensibly complex. There are good reasons also for it not to be crippled by lack of functionality. Are we stuck? There are two ways to deal with this problem. This note is a proposal that we adopt both of them. The reason the problem can be solved is that the sets of options that we have been considering (blocking or not, contiguous buffer or not, heterogeneous machines or not, etc.) are orthogonal, and one of a large number of possible send routines can thus be specified simply and without extra parameters. In addition, certain subsets of the set of all operations can be identified and packaged for ease of use. Here is a suggestion for how to organize the routines and end up with a powerful, flexible, yet easy-to-use library of point-to-point functions. (We assume that these will all be renamed to have an MPI prefix, to avoid name-space pollution) Level 4 (very restrictive, very simple): send(dest,tag,buf,len) recv(dest,tag,buf,len) Both of these block until the operation is locally complete. The buffer is a contiguous sequence of bytes. No special protocols or translation for heterogeneous machines. This level is enough to express lots of parallel algorithms. The main reason we need more is for efficiency and control. We now add these in steps. Level 3 (still simple, but allows for overlap of computation and communication and better buffer management, and allows heterogeneous machines to communicate.) bsend(tag,dest,bufadd,buflen) nsend(tag,dest,bufadd,buflen) bsendh(tag,dest,bufadd,datatype,numitems) nsendh(tag,dest,bufadd,datatype,numitems) brecv(tag,dest,bufadd,buflen) nrecv(tag,dest,bufadd,buflen) brecvh(tag,dest,bufadd,datatype,numitems) nrecvh(tag,dest,bufadd,datatype,numitems) Here the n or b selects block or nonblocking operations, h specifies translation for heterogeneous machines. The restrictions (which allow the parameter lists to be short and the number of routines to be small) are: no noncontiguous buffers, data in message all the same type, no globally synchronized, CSP-like send-receive, and no special protocol (like the ready-receiver protocol discussed at the last meeting). In order to emphasize the orthogonality, we might organize these routine names in a diagram like this: [n][send][ ] [b][recv][h] That is, one makes a choice in each column. Level 2 (access to almost all MPI functionality via a large number of orthogonally named routines): [c][n][send][ ][ ] [s][b][recv][h][rr] [g][s] the first column designates the buffer type (contiguous, strided, general gather/scatter), the second is the termination type (nonblocking, blocking, synchronized), then send or receive, "h" or not for heterogeneous communication, "rr" or not for "ready-receiver" protocol. This is a lot of routines (3 x 3 x 2 x 2 x 2 = 72) but because of orthogonality it is easy to understand. The only restriction here is that there are no "channels": the initialization of a send operation is combined with initiating it. Level 1 (full MPI functionality, in a small, flexible set of routines) handle = init_send(tag,dest) mod_send(handle,option,value) do_send(handle) free_send(handle) along with the corresponding receive routines. The idea is that the "init" routines will set up a usable set of defaults for all the options, specifying only minimal parameters, and then the "mod" routine can be repeatedly called to specify all the options with regard to buffer type, termination type, etc. The point is that at this level we can offer not only all possible options, with a small number of routines, and include the "setup" routines proposed by Marc for increased efficiency, but we also allow room for growth in the standard through the addition of more values for "option" in the "mod" calls. So here's how it might look, this time from the bottom up: Level 1: full functionality, small number of routines, channel setup Level 2: sequences of Level 1 calls of the form init... mod... mod... mod... do... are given specific names in an organized way. This promotes ease of use and also efficiency (fewer calls). Level 3: several of the Level 2 routines are renamed and promoted to this level to provide a useful subset of MPI functionality in a small number of routines, each having the minimal number of parameters. Level 4: ultra-simple interface All four levels would be part of the standard and it would be possible to mix levels in the same program. We have not considered carefully yet how this interacts with groups and contexts, but similar ideas might be useful there, where there are similar problems in providing both functionality and simplicity. Comments? Rusty Lusk and Bill Gropp From owner-mpi-comm@CS.UTK.EDU Wed Feb 3 06:35:54 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA01865; Wed, 3 Feb 93 06:35:54 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA23802; Wed, 3 Feb 93 06:35:00 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Wed, 3 Feb 1993 06:34:59 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from marge.meiko.com by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA23794; Wed, 3 Feb 93 06:34:56 -0500 Received: from hub.meiko.co.uk by marge.meiko.com with SMTP id AA13771 (5.65c/IDA-1.4.4 for ); Wed, 3 Feb 1993 06:34:53 -0500 Received: from float.co.uk (float.meiko.co.uk) by hub.meiko.co.uk (4.1/SMI-4.1) id AA26850; Wed, 3 Feb 93 11:34:48 GMT Date: Wed, 3 Feb 93 11:34:48 GMT From: jim@meiko.co.uk (James Cownie) Message-Id: <9302031134.AA26850@hub.meiko.co.uk> Received: by float.co.uk (5.0/SMI-SVR4) id AA03425; Wed, 3 Feb 93 11:33:11 GMT To: lusk@antares.mcs.anl.gov Cc: mpi-comm@cs.utk.edu In-Reply-To: Rusty Lusk's message of Wed, 03 Feb 93 00:40:59 CST <9302030641.AA23933@donner.mcs.anl.gov> Subject: A suggestion for a multi-level MPI Content-Length: 581 The fundamental approach seems fine to me (though Chuck makes some good points...) However there's at least one point to note :- 1) The Fortran binding issue suggests that in the heterogeneous cases there should be a routine for each data type, rather than having the data type as an argument. -- Jim James Cownie Meiko Limited Meiko Inc. 650 Aztec West Reservoir Place Bristol BS12 4SD 1601 Trapelo Road England Waltham MA 02154 Phone : +44 454 616171 +1 617 890 7676 FAX : +44 454 618188 +1 617 890 5042 E-Mail: jim@meiko.co.uk or jim@meiko.com From owner-mpi-comm@CS.UTK.EDU Wed Feb 3 07:27:22 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA11163; Wed, 3 Feb 93 07:27:22 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA25571; Wed, 3 Feb 93 07:26:12 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Wed, 3 Feb 1993 07:26:11 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from ifi.uio.no by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA25557; Wed, 3 Feb 93 07:26:08 -0500 Received: from byflak.ifi.uio.no by ifi.uio.no with SMTP id ; Wed, 3 Feb 1993 13:26:05 +0100 From: "\ystein Gran Larsen" Received: by byflak.ifi.uio.no ; Wed, 3 Feb 93 13:26:04 +0100 Date: Wed, 3 Feb 93 13:26:04 +0100 Message-Id: <9302031226.AA26562@byflak.ifi.uio.no> To: mpi-comm@cs.utk.edu Subject: data types and heterogeneity Hi, I have read the minutes from the MPI meeting, January 6-8 1993. The section on message data types mentiones the use of XDR as a protocol for converting types between languages and machines. The standardization of Scalable Coherent Interface (SCI) has lead to a proposed standard for "Shared-Data Formats Optimized for Scalable Coherent Interface Processors", P1596.5. [Working group P1596.5 is chaired by David V. James (dvj@apple.com)] This proposed standard may be of relevance to the MPI-initiative, when it comes to supporting heterogeneous machines. -{\O}ystein From owner-mpi-comm@CS.UTK.EDU Wed Feb 3 11:56:47 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA17274; Wed, 3 Feb 93 11:56:47 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA07365; Wed, 3 Feb 93 11:54:56 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Wed, 3 Feb 1993 11:54:55 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from Aurora.CS.MsState.Edu by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA07357; Wed, 3 Feb 93 11:54:53 -0500 Received: by Aurora.CS.MsState.Edu (4.1/6.0s-FWP); id AA01169; Wed, 3 Feb 93 10:54:51 CST Date: Wed, 3 Feb 93 10:54:51 CST From: Tony Skjellum Message-Id: <9302031654.AA01169@Aurora.CS.MsState.Edu> To: mpi-comm@cs.utk.edu, gran@ifi.uio.no Subject: Re: data types and heterogeneity XDR as currently implemented causes one function call per datum. This feature must be changed (loops moved inside fundamental converter routines) to provide for better performance (less bandwidth impact of converting data types). I am eager to see the report mentioned by Oystein. - Tony From owner-mpi-comm@CS.UTK.EDU Wed Feb 3 13:33:45 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA19846; Wed, 3 Feb 93 13:33:45 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA12323; Wed, 3 Feb 93 13:31:59 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Wed, 3 Feb 1993 13:31:58 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from iliamna.cse.ogi.edu by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA12315; Wed, 3 Feb 93 13:31:56 -0500 Received: by iliamna.cse.ogi.edu (/\==/\ Smail3.1.25.1 #25.15) id ; Wed, 3 Feb 93 10:31 PST Message-Id: Date: Wed, 3 Feb 93 10:31 PST From: otto@iliamna.cse.ogi.edu (Steve Otto) To: mpi-comm@cs.utk.edu Subject: Mix + Match, NOT At first look, I definitely like the proposal of Lusk and Gropp. It seems to be the right way to set up a hierarchy of the 1024+ routines that we are inventing. One thing that bothers me: at the last meeting it was said by several people that one would be allowed to mix+match different kinds of sends and recvs to each other. That is, I could do a "cnsend()" and have that consumed by a "sbrecvhrr()". I think we are asking for lots of trouble if the standard states that this is allowed. If it is allowed, an implementor has to worry about 72 x 72 = 5184 different types of behavior for pt-pt communication. I don't see a strong reason for allowing mix+match, and on the implementation side it seems to me that there are many reasons for wanting the restriction that a send of flavor 53 (out of a possible 72) should be matched by a recv of the same flavor, 53. It is true that it seems like many flavors could be mixed, but consider the synchronization choices. If we allow mixing, then the implementor of the sbrecv() routine has to insure that "sensible" behavior results no matter what send routine was used. As I understand it, this means the implementation cannot be done in an orthogonal manner. I'm not even sure we could define "sensible" for all 5184 combinations! Comments? Am I missing something? --Steve Otto From owner-mpi-comm@CS.UTK.EDU Wed Feb 3 18:43:47 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA29539; Wed, 3 Feb 93 18:43:47 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA27997; Wed, 3 Feb 93 18:41:34 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Wed, 3 Feb 1993 18:41:33 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from sol.cs.wmich.edu by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA27989; Wed, 3 Feb 93 18:41:32 -0500 Received: from id.wmich.edu by cs.wmich.edu (4.1/SMI-4.1) id AA14014; Wed, 3 Feb 93 18:35:11 EST Date: Wed, 3 Feb 93 18:35:11 EST From: john@cs.wmich.edu (John Kapenga) Message-Id: <9302032335.AA14014@cs.wmich.edu> To: mpi-comm@CS.UTK.EDU Subject: Re: A suggestion for a multi-level MPI James Cownie remarks on needing a seperate send routine for each data type in the Fortran Binding. For both the C and Fortran versions there could be seperate calls setting up the Data Addresses (no Data need be moved) and then a single send could take the value returned by the Data Address Setup Call. No system call required on the Data Setup Call. This might require that the array passed to the Data Setup Call not be modified before the send call is made, to avoid recopying it. This also allows later extensions to other Data Addressing modes, without changing the send/recieve syntax. This reduces the number of calls by almost a factor of three, uncouples the data location from the send/recieve syntax and allows arguement types on all calls to be correct and fixed. It does make a bit more processing (trival) and the need for the user to mamage the Data Setup return value and not modify the address array (two of its bad points). john From owner-mpi-comm@CS.UTK.EDU Wed Feb 3 23:45:54 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA02591; Wed, 3 Feb 93 23:45:54 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA09607; Wed, 3 Feb 93 23:43:48 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Wed, 3 Feb 1993 23:43:47 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from Aurora.CS.MsState.Edu by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA09599; Wed, 3 Feb 93 23:43:45 -0500 Received: by Aurora.CS.MsState.Edu (4.1/6.0s-FWP); id AA00850; Wed, 3 Feb 93 22:43:44 CST Date: Wed, 3 Feb 93 22:43:44 CST From: Tony Skjellum Message-Id: <9302040443.AA00850@Aurora.CS.MsState.Edu> To: mpi-comm@cs.utk.edu Subject: Anonymous ftp site aurora.cs.msstate.edu A paper of relevance to the MPI project, describing "Zipcode 1.0" is available by anonymous FTP, in the pub/reports subdirectory: zipcode_sep92.ps.Z The site is aurora.cs.msstate.edu. - Tony Skjellum From owner-mpi-comm@CS.UTK.EDU Thu Feb 4 00:02:32 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA02797; Thu, 4 Feb 93 00:02:32 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA10374; Thu, 4 Feb 93 00:01:41 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Thu, 4 Feb 1993 00:01:39 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from antares.mcs.anl.gov by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA10364; Thu, 4 Feb 93 00:01:38 -0500 Received: from donner.mcs.anl.gov by antares.mcs.anl.gov (4.1/SMI-GAR) id AA08517; Wed, 3 Feb 93 23:01:36 CST Received: by donner.mcs.anl.gov (4.1/GCF-5.8) id AA26443; Wed, 3 Feb 93 23:01:34 CST Message-Id: <9302040501.AA26443@donner.mcs.anl.gov> To: mpi-comm@cs.utk.edu Subject: Multiple levels Date: Wed, 03 Feb 93 23:01:33 CST From: Rusty Lusk Chuck -- Thanks for your comments. They are of two kinds. The first kind are about the suggested multiple-level organization of the library. There I think I didn't explain some things well enough, and will try to do better here. The other kind are about what should be in or out of the library altogether, and there I can only try to summarize what people talked about in Dallas; the things we incorporated are just the things that the straw votes indicated people wanted in the standard. | | I do view the interface as being layered, but I would organize the layers | differently. And I have a couple of arguments that I haven't tried out | yet against some of the send/recv routines. | | First, it sounds to me like you are suggesting that the 72 routines | be implemented as 72 simple routines that each make a small number of | subroutine calls to very simple routines. This is not at all what we intended to suggest. We used the word "levels" rather than "layers". The upper levels could be *defined* in terms of the lower ones (That's what makes them upper-level) but the intention is that they would *not* be implemented as calls to the lower-level routines. In fact what we called level 2 (large number of routines) exists partly to avoid the multiple subroutine calls necessary in level 1 (where there are a small number of routines.) In fact, level 2 routines don't need to use the data structures implied by the get_handle-mod_handle-use_handle-free_handle at all. They just have to do what they say they do, as efficiently as possible. | | For heterogeneity, I would argue that it is wrong to put the datatype | translation filter in the communications channel. | Our discussions seemed to indicate that people want it there. The popularity of systems like PVM, p4, Express, and others that handle this for the user attests to this. I think the reason is portability; people like being able to move code unchanged from a heterogeneous network to a multicomputer and back. The system can (relatively easily) handle all the translations, including figuring out whether it is necessary at all or not. Many users hardly even know about the non-portability of data formats; they think (in my opinion rightly) that the communication system should handle that if they want it to. | In your model, it | is relatively easy to do things like: | | bsendh (dest, tag, buf, FLOAT, 1); | or even: | bsendh (dest, tag, buf, ARRAY|FLOAT, nitems); | but what about | bsendh (dest, tag, buf, rpc_t, 1); | where 'rpc_t' is some user defined data structure? Thus, this model | lacks generality. Not quite true, although not entirely false. In the "draft implementation" of mpi that you can get from here, we use an iovec-like structure which is a vector of (addr,datatype,nitems) triples to describe a scattered, mixed type buffer. An arbitrary structure can be sent this way, provided the user builds the structure descriptor (It's too bad compilers don't do this for us.) | | It would be as efficient to implement the translation routines completely | separate from the communications routines. | .... | | The above is essentially the same argument that I'm going to use for | strided messages. Strided messages are only efficient if you have | hardware that can DMA the strided vector, and then its not so efficient | that it's obviously worth the effort. Machines do have this kind of hardware, and we can expect to see more of it. One thing that I will continue to argue in favor of is giving the user access to as much of the speed the hardware can deliver as possible. | | For ready-receiver, I'm going to argue that the communications subsystem | should always assume the receiver is ready and then handle the case where | this isn't true. | Some existing machines can perform *significantly* faster if the user can relieve their system of some buffer-management chores. If we don't give the power users access to this efficiency, they simply won't use MPI. And it will come to pass that everyone will agree that to get real efficiency, you have to use the vendor's unique routines. Or you have to use "MPI with the XXX extension package". Portability will be destroyed. This is a way the MPI effort could fail. | | Implementing the ready-receiver places strong semantics on the sender | and receiver. The sender has to know whether or not the receiver was | implemented in a fashion that allows the sender to assume that the receiver | is ready. | Agreed. The naive user can use what we call the Level 3 or Level 4 routines, and never read the chapter in the manual that describes "ready-receiver" semantics and requirements. The power user can use them to write *portable* fast code. | | Finally, let's consider synchronous messages versus blocking messages. For | most applications, you want reliability. Implementing blocking | non-synchronous messages doesn't help you implement this. You can't reuse | the buffer being sent until you know that it has been received at the other | side. So you might as well either send the message synchronously, or send | it non-blocking and go off and do something else while you're waiting to see | if you'll need to retransmit the buffer. | Most people want to compute while the message is in transit, and even while the buffer is being emptied. They don't want this capability taken from them. | | This reduces the number of interface routines to 8. I could probably | be convinced that adding 4 more interface routines to support strided | messages doesn't overly complicate the interface. | The Level 4 routines are only 2. More routines are needed only for more efficiency and functionality. In the last few weeks I have talked to lots of people about the MPI effort, and they are mainly afraid that we will leave something important out. If we don't supply all the functionality and efficiency that we possibly can, MPI will fail. The problem is to do so without making the package too complicated. Our posting about multiple levels was merely an attempt to demonstrate that it could be done. Regards, Rusty Lusk & Bill Gropp From owner-mpi-comm@CS.UTK.EDU Thu Feb 4 01:30:19 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA04177; Thu, 4 Feb 93 01:30:19 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA13427; Thu, 4 Feb 93 01:29:10 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Thu, 4 Feb 1993 01:29:09 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from msr.EPM.ORNL.GOV by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA13419; Thu, 4 Feb 93 01:29:08 -0500 Received: by msr.EPM.ORNL.GOV (5.67/1.34) id AA12505; Thu, 4 Feb 93 01:29:05 -0500 Date: Thu, 4 Feb 93 01:29:05 -0500 From: geist@msr.EPM.ORNL.GOV (Al Geist) Message-Id: <9302040629.AA12505@msr.EPM.ORNL.GOV> To: mpi-comm@cs.utk.edu Subject: Re: 5184 bottles of sends on the wall... Steve Otto proposes that the 72 (or 648 if each data type is separate) sends be matched to the same type of recv. While I agree with Steve that we will not be able to define much less test the 5184 combinations if we allow otherwise, There are obvious cases where the user would want to mismatch send/recv For example, she may have to do a blocking send, but can get away with a non-blocking recv. Not all combinations make sense, but we can't restrict the user to no mismatches. Al Geist From owner-mpi-comm@CS.UTK.EDU Thu Feb 4 02:10:22 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA05537; Thu, 4 Feb 93 02:10:22 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA14590; Thu, 4 Feb 93 02:08:44 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Thu, 4 Feb 1993 02:08:43 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from msr.EPM.ORNL.GOV by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA14582; Thu, 4 Feb 93 02:08:41 -0500 Received: by msr.EPM.ORNL.GOV (5.67/1.34) id AA12583; Thu, 4 Feb 93 02:08:39 -0500 Date: Thu, 4 Feb 93 02:08:39 -0500 From: geist@msr.EPM.ORNL.GOV (Al Geist) Message-Id: <9302040708.AA12583@msr.EPM.ORNL.GOV> To: mpi-comm@cs.utk.edu Subject: Re: Rusty and Bill ride again... Whoa there Rusty and Bill. Let's not play to fast and loose with the facts. >Most people want to compute even while the buffer is being emptied... Actually most people don't want to think or know about how messages are sent they just want it reliable and fast. Only high priests are concerned with the pain of non-blocking send. >If we don't supply all the functionality that we possibly can, >MPI will fail. The problem is to do so without making the package >too complicated. The complication I am seeing in MPI will also make it fail. 72 sends?? Appears to violate several of our goals, like similar to existing packages. small set of routines. >The naive user never has to read the chapter in the manual... I hope the user doesn't have to make room on his shelves for volumes of the MPI manual. Al From owner-mpi-comm@CS.UTK.EDU Thu Feb 4 02:41:49 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA05708; Thu, 4 Feb 93 02:41:49 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA15558; Thu, 4 Feb 93 02:40:34 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Thu, 4 Feb 1993 02:40:32 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from gmdzi.gmd.de by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA15544; Thu, 4 Feb 93 02:40:30 -0500 Received: from f1neuman.gmd.de (f1neuman) by gmdzi.gmd.de with SMTP id AA08675 (5.65c/IDA-1.4.4 for ); Thu, 4 Feb 1993 08:39:03 +0100 Received: by f1neuman.gmd.de id AA14246; Thu, 4 Feb 1993 08:38:42 +0100 Date: Thu, 4 Feb 1993 08:38:42 +0100 From: Rolf.Hempel@gmd.de Message-Id: <9302040738.AA14246@f1neuman.gmd.de> To: mpi-comm@cs.utk.edu Subject: Re: 5184 bottles of sends on the wall... Cc: gmap10@f1neuman.gmd.de I was glad to read Steve's proposal not to allow mismatches between different types of sends and receives. If we don't follow that line, we will most likely cause many problems to the implementors of MPI which we cannot forsee at the moment. Also, the requirement of matching sends and receives could be used to check the correct behaviour of the user program in some debugging mode. Al's counter-example of course makes sense, but if you have to use a blocking send, you can get the same by using the non-blocking version followed directly by the wait-for-completion. I'm looking forward to seeing an example where you really need the mismatch. Rolf From owner-mpi-comm@CS.UTK.EDU Thu Feb 4 03:16:46 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA05912; Thu, 4 Feb 93 03:16:46 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA17215; Thu, 4 Feb 93 03:14:56 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Thu, 4 Feb 1993 03:14:53 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from sol.cs.wmich.edu by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA17205; Thu, 4 Feb 93 03:14:51 -0500 Received: from id.wmich.edu by cs.wmich.edu (4.1/SMI-4.1) id AA15627; Thu, 4 Feb 93 03:08:28 EST Date: Thu, 4 Feb 93 03:08:28 EST From: john@cs.wmich.edu (John Kapenga) Message-Id: <9302040808.AA15627@cs.wmich.edu> To: mpi-comm@cs.utk.edu Subject: Re: 5184 bottles of sends on the wall... Rolf asks -- > ... > followed directly by the wait-for-completion. I'm looking forward to > seeing an example where you really need the mismatch. > > Rolf By using the wait_for_completion you never need a blocking send/receive operation. The code looks a bit ugly. Right now we have to have a "mismatch" because we don't have a synchronized send/receive pair ;-). Actually I thought the reason that the synchronized send/recieve pair was there initially was to help insure correctness in matching calls, as well as a possible simpler implementation. john From owner-mpi-comm@CS.UTK.EDU Thu Feb 4 07:03:39 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA07047; Thu, 4 Feb 93 07:03:39 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA01249; Thu, 4 Feb 93 07:02:33 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Thu, 4 Feb 1993 07:02:32 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from marge.meiko.com by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA01241; Thu, 4 Feb 93 07:02:28 -0500 Received: from hub.meiko.co.uk by marge.meiko.com with SMTP id AA09794 (5.65c/IDA-1.4.4 for ); Thu, 4 Feb 1993 07:02:22 -0500 Received: from float.co.uk (float.meiko.co.uk) by hub.meiko.co.uk (4.1/SMI-4.1) id AA02743; Thu, 4 Feb 93 12:02:17 GMT Date: Thu, 4 Feb 93 12:02:17 GMT From: jim@meiko.co.uk (James Cownie) Message-Id: <9302041202.AA02743@hub.meiko.co.uk> Received: by float.co.uk (5.0/SMI-SVR4) id AA03570; Thu, 4 Feb 93 12:00:32 GMT To: otto@iliamna.cse.ogi.edu Cc: mpi-comm@cs.utk.edu In-Reply-To: Steve Otto's message of Wed, 3 Feb 93 10:31 PST Subject: ~ (Mix + Match, NOT) Content-Length: 2480 The only place where I can see a need for matching of send and receive formats is in the heterogeneous case. Here I would be happy to insist that a heterogeneous send be matched to a heterogeneous receive. Since I don't want to enforce in MPI the implemntation decision of whether the data type conversion occurs at sender or receiver I need the data type information in both places. I can see NO reason to restrict any other combinations. Discussion ---------- There are two major areas which cause the many sends 1) layout of data in store. (contiguous, strided, by iovec) 2) synchronisation behaviour (return asap, return when buffer free, return when data received at other end) Data layout ----------- Data layout is an issue which is entirely local. Why does it matter to the receiver how the sender chose to layout her buffer (or the converse) ? This information hiding is one of the things which is an advantage of message passing. The data is necessarily serialised to some degree over the communications medium, so where's the problem ? I believe it is actually HARDER to insist that the buffer sepcifications match at each end than not to do this. (Certainly I have to pass additional information with the message if I'm to check this.) Synchronisation behaviour ------------------------- Once again this seems to me to be largely to do with local issues of buffer management. (Though of course my algorithm may need certain synchronisation behaviour, in which case I must code it using the correct forms of send and recv at both ends.) However I can see no reason to INSIST that the same style of send and receive is used. Indeed there are strong reasons NOT to do so. Consider (for example) a server with multiple clients. I'd certainly like to write it like this :-- Set up lots of non-blocking receives while (running) { wait for a recv service the request get a reply buffer (may need to wait for one) queue a reply requeue the receive buffer } However the clients can simply make requests thus blocking send request blocking receive reply Here is an immediate mismatch in the synchronisations. -- Jim James Cownie Meiko Limited Meiko Inc. 650 Aztec West Reservoir Place Bristol BS12 4SD 1601 Trapelo Road England Waltham MA 02154 Phone : +44 454 616171 +1 617 890 7676 FAX : +44 454 618188 +1 617 890 5042 E-Mail: jim@meiko.co.uk or jim@meiko.com From owner-mpi-comm@CS.UTK.EDU Thu Feb 4 09:44:02 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA11973; Thu, 4 Feb 93 09:44:02 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA07037; Thu, 4 Feb 93 09:42:34 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Thu, 4 Feb 1993 09:42:33 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from [132.175.13.2] by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA07029; Thu, 4 Feb 93 09:42:30 -0500 Received: from panther.cs.sandia.gov by cs.sandia.gov (4.1/SMI-4.1) id AA12554; Thu, 4 Feb 93 07:42:27 MST Received: by panther.cs.sandia.gov (Smail3.1.28.1 #1) id m0nK7mc-0016ZKC; Thu, 4 Feb 93 07:42 MST Message-Id: Date: Thu, 4 Feb 93 07:42 MST From: srwheat@cs.sandia.gov (Stephen R. Wheat) To: mpi-comm@cs.utk.edu Subject: Re: Rusty and Bill ride again... # Whoa there Rusty and Bill. Let's not play to fast and loose with the facts. # >Most people want to compute even while the buffer is being emptied... # Actually most people don't want to think or know about how messages # are sent they just want it reliable and fast. # Only high priests are concerned with the pain of non-blocking send. Not so; we have those that are not only concerned about being able to compute while messages are in transit, but also wanting to assure that minimal system space is required. That is, they want to manage memory much more tightly than a "buffered" system allows. We have a lot of applications types that find it very hard to fit an application into memory and still allocate sufficient message buffers. By the way, paging is not a viable solution :-). Stephen From owner-mpi-comm@CS.UTK.EDU Thu Feb 4 09:59:26 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA12166; Thu, 4 Feb 93 09:59:26 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA07696; Thu, 4 Feb 93 09:58:25 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Thu, 4 Feb 1993 09:58:23 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from [132.175.13.2] by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA07688; Thu, 4 Feb 93 09:58:22 -0500 Received: from panther.cs.sandia.gov by cs.sandia.gov (4.1/SMI-4.1) id AA12746; Thu, 4 Feb 93 07:58:18 MST Received: by panther.cs.sandia.gov (Smail3.1.28.1 #1) id m0nK81y-0016ZKC; Thu, 4 Feb 93 07:58 MST Message-Id: Date: Thu, 4 Feb 93 07:58 MST From: srwheat@cs.sandia.gov (Stephen R. Wheat) To: john@cs.wmich.edu, mpi-comm@cs.utk.edu Subject: Re: 5184 bottles of sends on the wall... as an implementor of blocked and non-blocked messaging systems, I have not experienced any "system" problems associated with matching mixed send/receives. While worrying about the "potential" implementation difficulties, I fear that we make use of the system so difficult that it won't be used (so would that make it easy to use?). Furthermore, as for the theme that seems to prevail concerning features that only "wizards" would use, if we do not provide wizardly type features, then the wizards will not use MPI. Then to whom will the "novice" types refer when they finally see the light of low-performance codes running in a high-performance compute environment? Stephen From owner-mpi-comm@CS.UTK.EDU Thu Feb 4 10:28:11 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA12742; Thu, 4 Feb 93 10:28:11 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA09131; Thu, 4 Feb 93 10:26:01 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Thu, 4 Feb 1993 10:26:00 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from Aurora.CS.MsState.Edu by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA09123; Thu, 4 Feb 93 10:25:58 -0500 Received: by Aurora.CS.MsState.Edu (4.1/6.0s-FWP); id AA01127; Thu, 4 Feb 93 09:25:56 CST Date: Thu, 4 Feb 93 09:25:56 CST From: Tony Skjellum Message-Id: <9302041525.AA01127@Aurora.CS.MsState.Edu> To: mpi-comm@cs.utk.edu, john@cs.wmich.edu Subject: Re: 5184 bottles of sends on the wall... Please remember that overlapped send/receive will only make sense on systems with excess memory bandwidth. Hence, it might make more sense to implement a system where this is permitted by the implementor, but not required. Otherwise, calculations might slow down instead of achieving pipelined improvements. As Al points out, it is only the heavy-duty guys who try this sort of stuff... - Tony From owner-mpi-comm@CS.UTK.EDU Thu Feb 4 10:30:25 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA12830; Thu, 4 Feb 93 10:30:25 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA09278; Thu, 4 Feb 93 10:28:57 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Thu, 4 Feb 1993 10:28:56 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from sol.cs.wmich.edu by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA09270; Thu, 4 Feb 93 10:28:53 -0500 Received: from id.wmich.edu by cs.wmich.edu (4.1/SMI-4.1) id AA16295; Thu, 4 Feb 93 10:22:30 EST Date: Thu, 4 Feb 93 10:22:30 EST From: john@cs.wmich.edu (John Kapenga) Message-Id: <9302041522.AA16295@cs.wmich.edu> To: mpi-comm@CS.UTK.EDU Subject: Re: 5184 bottles of sends on the wall... My initial concept of mixing IO types was that blocking and non-blocking could be freely mixed, but that synchronized was special and a synchronized send would only match a synchronized receive. The rationale for dropping the synchronized recieve is OK with me. I have no problems with the current proposal. Overlappping communication and computation is the only way to get good prerformance in many applications. It must be assumed to be the norm rather than the exception. I would not be able to use a system that did not support it. john From owner-mpi-comm@CS.UTK.EDU Thu Feb 4 10:46:15 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA13213; Thu, 4 Feb 93 10:46:15 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA10063; Thu, 4 Feb 93 10:45:12 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Thu, 4 Feb 1993 10:45:11 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from marge.meiko.com by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA10042; Thu, 4 Feb 93 10:45:06 -0500 Received: from hub.meiko.co.uk by marge.meiko.com with SMTP id AA12579 (5.65c/IDA-1.4.4 for ); Thu, 4 Feb 1993 10:45:01 -0500 Received: from float.co.uk (float.meiko.co.uk) by hub.meiko.co.uk (4.1/SMI-4.1) id AA06100; Thu, 4 Feb 93 15:44:56 GMT Date: Thu, 4 Feb 93 15:44:56 GMT From: jim@meiko.co.uk (James Cownie) Message-Id: <9302041544.AA06100@hub.meiko.co.uk> Received: by float.co.uk (5.0/SMI-SVR4) id AA03785; Thu, 4 Feb 93 15:43:15 GMT To: tony@Aurora.CS.MsState.Edu Cc: mpi-comm@cs.utk.edu, john@cs.wmich.edu In-Reply-To: Tony Skjellum's message of Thu, 4 Feb 93 09:25:56 CST <9302041525.AA01127@Aurora.CS.MsState.Edu> Subject: 5184 bottles of sends on the wall... Content-Length: 1025 > Please remember that overlapped send/receive will only make sense on > systems with excess memory bandwidth. Hence, it might make more sense > to implement a system where this is permitted by the implementor, but > not required. Otherwise, calculations might slow down instead of > achieving pipelined improvements. True, and all the more reason to avoid as many copies as possible. After all if taking it from the user buffer consumes bandwidth 1, then the bandwidth cost of a send which needed a copy will be 2+1 = 3. This can be significant. (Of course I'm assuming a "fair" memory system here, in which taking from one port only loses the same from another. I've seen systems where this wasn't true, and this can change the balance.) -- Jim James Cownie Meiko Limited Meiko Inc. 650 Aztec West Reservoir Place Bristol BS12 4SD 1601 Trapelo Road England Waltham MA 02154 Phone : +44 454 616171 +1 617 890 7676 FAX : +44 454 618188 +1 617 890 5042 E-Mail: jim@meiko.co.uk or jim@meiko.com From owner-mpi-comm@CS.UTK.EDU Thu Feb 4 15:33:00 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA19567; Thu, 4 Feb 93 15:33:00 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA25767; Thu, 4 Feb 93 15:30:40 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Thu, 4 Feb 1993 15:30:37 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from fslg8.fsl.noaa.gov by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA25752; Thu, 4 Feb 93 15:30:34 -0500 Received: by fslg8.fsl.noaa.gov (5.57/Ultrix3.0-C) id AA04724; Thu, 4 Feb 93 20:30:30 GMT Received: by macaw.fsl.noaa.gov (4.1/SMI-4.1) id AA05190; Thu, 4 Feb 93 13:29:24 MST Date: Thu, 4 Feb 93 13:29:24 MST From: hender@macaw.fsl.noaa.gov (Tom Henderson) Message-Id: <9302042029.AA05190@macaw.fsl.noaa.gov> To: mpi-comm@cs.utk.edu Subject: Re: A suggestion for a multi-level MPI I especially like "Level 1" of Rusty and Bill's proposal. I would like to see more detail in this section, like examples of mod_xxx() and do_xxx() routines. I think there will be a good deal of support for Level 1 with all the talk that's been going on about handles/invoices/envelopes (of course there's that annoying syntax thing...). I am concerned about how heterogeneous communications are handled (or avoided) in Levels 3 and 4. I would be a lot more comfortable if ALL send() and recv() routines were heterogeneous. Here's a slight modification to the original proposal: Level 4 ("novice users"): send(dest,tag,bufadd,datatype,numitems) recv(dest,tag,bufadd,datatype,numitems) Level 3 ("dangerous users" [like me :) ]): bsend(tag,dest,bufadd,datatype,numitems) nsend(tag,dest,bufadd,datatype,numitems) brecv(tag,dest,bufadd,datatype,numitems) nrecv(tag,dest,bufadd,datatype,numitems) or [n][send] [b][recv] in Bill and Rusty's shorthand. Level 2 ("really frightening users"): [c][n][send][ ] [s][b][recv][rr] [g][s] Here's the arguments: - I don't like the idea of having to tell novice users that they must change all the MPI communication routine calls in their code before their program will run on a heterogeneous system. I think a lot of novice users will be using heterogeneous workstation networks. The current proposal forces these folks to learn at least two versions of send() and recv(). - I don't think there are any significant performance reasons for having separate heterogeneous versions of these calls, as long as we require a receiving process to know what incoming message contents will be (which seems to be consistent with the original proposal). - I don't think that the additional "datatype" argument reduces "ease of use" very much. An incorrect datatype should be no more difficult to debug than an incorrect "bufadd" or "buflen" (and should be easier than a bad "tag" or "dest", especially if deadlock results!). In FORTRAN77, "numitems" should be easier to code portably than "buflen". Even in C, there is little difference in complexity between portable calls to heterogeneous and non-heterogeneous versions: (non-heterogeneous) float mybuf[NUMITEMS]; ... send(dest, tag, mybuf, NUMITEMS * sizeof(float)); (heterogeneous) send(dest, tag, mybuf, MPI_FLOAT, NUMITEMS); - This gets rid of a whole bunch of send() and recv() variant routines (at least 22). - If we really want non-heterogeneous communication, it could be stuck in Level 1 with all the other stuff that's really dangerous for "novices". Another option would be to allow a MPI_BYTE datatype (not sure how this would work in FORTRAN). This may be useful for library developers. I think it's a good idea to keep "raw bytes" away from novice users (especially in FORTRAN). Tom Henderson hender@fsl.noaa.gov From owner-mpi-comm@CS.UTK.EDU Thu Feb 4 15:49:14 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA20005; Thu, 4 Feb 93 15:49:14 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA27824; Thu, 4 Feb 93 15:48:00 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Thu, 4 Feb 1993 15:47:58 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from Aurora.CS.MsState.Edu by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA27816; Thu, 4 Feb 93 15:47:57 -0500 Received: by Aurora.CS.MsState.Edu (4.1/6.0s-FWP); id AA01577; Thu, 4 Feb 93 14:47:56 CST Date: Thu, 4 Feb 93 14:47:56 CST From: Tony Skjellum Message-Id: <9302042047.AA01577@Aurora.CS.MsState.Edu> To: mpi-comm@cs.utk.edu, hender@macaw.fsl.noaa.gov Subject: Re: A suggestion for a multi-level MPI Tom, It is an artificial requirement to restrict heterogeneous messages to a single data type. - Tony From owner-mpi-comm@CS.UTK.EDU Thu Feb 4 15:53:51 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA20074; Thu, 4 Feb 93 15:53:51 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA28380; Thu, 4 Feb 93 15:52:40 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Thu, 4 Feb 1993 15:52:38 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from fslg8.fsl.noaa.gov by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA28367; Thu, 4 Feb 93 15:52:36 -0500 Received: by fslg8.fsl.noaa.gov (5.57/Ultrix3.0-C) id AA04817; Thu, 4 Feb 93 20:52:32 GMT Received: by macaw.fsl.noaa.gov (4.1/SMI-4.1) id AA05241; Thu, 4 Feb 93 13:51:27 MST Date: Thu, 4 Feb 93 13:51:27 MST From: hender@macaw.fsl.noaa.gov (Tom Henderson) Message-Id: <9302042051.AA05241@macaw.fsl.noaa.gov> To: mpi-comm@cs.utk.edu Subject: Re: A suggestion for a multi-level MPI > Tom, > > It is an artificial requirement to restrict heterogeneous messages to a single > data type. > - Tony > I agree. My interpretation of Rusty and Bill's proposal is that messages that contain multiple data types would be constructed using the Level 1 routines-- something similar to Zipcode's invoices. Tom From owner-mpi-comm@CS.UTK.EDU Thu Feb 4 17:02:43 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA21865; Thu, 4 Feb 93 17:02:43 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA03500; Thu, 4 Feb 93 17:01:23 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Thu, 4 Feb 1993 17:01:22 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from iliamna.cse.ogi.edu by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA03492; Thu, 4 Feb 93 17:01:20 -0500 Received: by iliamna.cse.ogi.edu (/\==/\ Smail3.1.25.1 #25.15) id ; Thu, 4 Feb 93 14:01 PST Message-Id: Date: Thu, 4 Feb 93 14:01 PST From: otto@iliamna.cse.ogi.edu (Steve Otto) To: mpi-comm@cs.utk.edu Subject: 5184; one small point I got into my office late today, and I must say, how exciting to see all the mail concerning 5184! Jim Cownie writes: >The data is necessarily serialised to some degree over the >communications medium, so where's the problem ? I believe it is >actually HARDER to insist that the buffer sepcifications match at each >end than not to do this. (Certainly I have to pass additional >information with the message if I'm to check this.) I did not mean to imply that the standard should RULE OUT mix+match or that it should check for it at run time, merely that the standard should not make the guarantee that one IS allowed to do this. There is an important difference. I really don't believe we can foresee everything, so we should be conservative on what the standard guarantees. Jim agrees that in the case of heterogeneity, it does seem that we want matching: >The only place where I can see a need for matching of send and receive >formats is in the heterogeneous case. Here I would be happy to insist >that a heterogeneous send be matched to a heterogeneous receive. So maybe we don't want to allow all 5184 combinations. OK, at least we are admitting that there is an issue here. --Steve Otto From owner-mpi-comm@CS.UTK.EDU Thu Feb 4 17:52:12 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA23307; Thu, 4 Feb 93 17:52:12 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA06478; Thu, 4 Feb 93 17:50:57 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Thu, 4 Feb 1993 17:50:56 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from ssdintel.ssd.intel.com by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA06466; Thu, 4 Feb 93 17:50:53 -0500 Received: from fuji.ssd.intel.com by SSD.intel.com (4.1/SMI-4.1) id AA17493; Thu, 4 Feb 93 14:50:39 PST Message-Id: <9302042250.AA17493@SSD.intel.com> To: otto@iliamna.cse.ogi.edu (Steve Otto) Cc: mpi-comm@cs.utk.edu, prp@SSD.intel.com Subject: Re: 5184; one small point In-Reply-To: Your message of "Thu, 04 Feb 93 14:01:00 PST." Date: Thu, 04 Feb 93 14:50:38 -0800 From: prp@SSD.intel.com > >The data is necessarily serialised to some degree over the > >communications medium, so where's the problem ? I believe it is > >actually HARDER to insist that the buffer sepcifications match at each > >end than not to do this. (Certainly I have to pass additional > >information with the message if I'm to check this.) > > I did not mean to imply that the standard should RULE OUT > mix+match or that it should check for it at run time, > merely that the standard should not make the > guarantee that one IS allowed to do this. There is an > important difference. I really don't believe we can foresee > everything, so we should be conservative on what the standard > guarantees. > > --Steve Otto > One of the important differences is that if the standard does not guarantee that you can mix, you can't write a portable program that mixes. Since we have seen one or two good examples of useful mixing, I think its important to guarantee the ability to mix at least for some combinations. My preference would be to guarantee mixing in all but specific cases, as few as possible. In fact, I would recommend against including features that cannot be mixed. I am optimistic that we can take care of the heterogeneous case in a sufficiently uniform way that mixing heterogeneous levels with non-heterogeneous becomes a non-issue. There was a good argument for not having a non-heterogeneous level, that would take care of it. The one case where mixing probably can't be justified is synchronous operation, where the send does not complete until the receive does. I've never liked this option, and worry that some support comes from misunderstanding about the difference between blocking and synchronous, as evidenced by Chuck's comment "You can't reuse the buffer being sent until you know that it has been received at the other side". This is of course not true. If the blocking send semantics are that 1) the send completes when the send buffer can be written safely and 2) message delivery is guaranteed, then you may happily overwrite the send buffer when the send completes with the assurance that the undamaged message will eventually appear at the receiver if its not already there. The only interesting difference between blocking send and synchronous send is in the amount of system buffering that may be required by applications. A correct application using blocking send might require some amount of system buffering, while a correct application that exclusively uses synchronous send requires no system buffering. In my opinion, we should focus on the problem of implied system buffering in the system environment area and eliminate synchronous send from the point-to-point portion of the standard. With reasonable care, we can make a clear and simple guarantee that appropriate different sends and receives can be mixed freely. This will allow people to write portable applications which use mixed structures. Paul From owner-mpi-comm@CS.UTK.EDU Thu Feb 4 17:56:31 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA23412; Thu, 4 Feb 93 17:56:31 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA06726; Thu, 4 Feb 93 17:55:42 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Thu, 4 Feb 1993 17:55:40 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from hub.ucsb.edu by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA06712; Thu, 4 Feb 93 17:55:37 -0500 Received: from garuda (garuda.ucsb.edu) by hub.ucsb.edu; id AA06657 sendmail 4.1/UCSB-2.0-sun Thu, 4 Feb 93 14:55:27 PST for mpi-comm@cs.utk.edu Received: by garuda (4.1/UCSB-v2) id AA22402; Thu, 4 Feb 93 13:56:59 PST Date: Thu, 4 Feb 93 13:56:59 PST From: ambuj%cs@hub.ucsb.edu (Ambuj Singh) Message-Id: <9302042156.AA22402@garuda> To: mpi-comm@cs.utk.edu Subject: Asynchronous send operations Hi: Here is a suggestion regarding the various kinds of send operations. From a semantic point of view, the synchronous and the asynchronous versions are meaningful. The further division of the asynchronous send into two kinds based on whether the buffer has been emptied or not is an implementation issue. Implementors of a system where buffer space is not a problem may wish to return without waiting for the buffer to empty whereas implementors of a system with limited buffer space may wish to suspend the operation until the buffer space is emptied. As to which version of asynchronous send operation is being implemented should be transparent and should not affect the correctness of a program. Ambuj Singh Dept. of CS Univ. of California Santa Barbara, CA 93106 ambuj@cs.ucsb.edu From owner-mpi-comm@CS.UTK.EDU Thu Feb 4 18:06:52 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA23575; Thu, 4 Feb 93 18:06:52 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA06726; Thu, 4 Feb 93 17:55:42 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Thu, 4 Feb 1993 17:55:40 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from hub.ucsb.edu by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA06712; Thu, 4 Feb 93 17:55:37 -0500 Received: from garuda (garuda.ucsb.edu) by hub.ucsb.edu; id AA06657 sendmail 4.1/UCSB-2.0-sun Thu, 4 Feb 93 14:55:27 PST for mpi-comm@cs.utk.edu Received: by garuda (4.1/UCSB-v2) id AA22402; Thu, 4 Feb 93 13:56:59 PST Date: Thu, 4 Feb 93 13:56:59 PST From: ambuj%cs@hub.ucsb.edu (Ambuj Singh) Message-Id: <9302042156.AA22402@garuda> To: mpi-comm@cs.utk.edu Subject: Asynchronous send operations Hi: Here is a suggestion regarding the various kinds of send operations. From a semantic point of view, the synchronous and the asynchronous versions are meaningful. The further division of the asynchronous send into two kinds based on whether the buffer has been emptied or not is an implementation issue. Implementors of a system where buffer space is not a problem may wish to return without waiting for the buffer to empty whereas implementors of a system with limited buffer space may wish to suspend the operation until the buffer space is emptied. As to which version of asynchronous send operation is being implemented should be transparent and should not affect the correctness of a program. Ambuj Singh Dept. of CS Univ. of California Santa Barbara, CA 93106 ambuj@cs.ucsb.edu From owner-mpi-comm@CS.UTK.EDU Thu Feb 4 18:15:06 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA23912; Thu, 4 Feb 93 18:15:06 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA06726; Thu, 4 Feb 93 17:55:42 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Thu, 4 Feb 1993 17:55:40 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from hub.ucsb.edu by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA06712; Thu, 4 Feb 93 17:55:37 -0500 Received: from garuda (garuda.ucsb.edu) by hub.ucsb.edu; id AA06657 sendmail 4.1/UCSB-2.0-sun Thu, 4 Feb 93 14:55:27 PST for mpi-comm@cs.utk.edu Received: by garuda (4.1/UCSB-v2) id AA22402; Thu, 4 Feb 93 13:56:59 PST Date: Thu, 4 Feb 93 13:56:59 PST From: ambuj%cs@hub.ucsb.edu (Ambuj Singh) Message-Id: <9302042156.AA22402@garuda> To: mpi-comm@cs.utk.edu Subject: Asynchronous send operations Hi: Here is a suggestion regarding the various kinds of send operations. From a semantic point of view, the synchronous and the asynchronous versions are meaningful. The further division of the asynchronous send into two kinds based on whether the buffer has been emptied or not is an implementation issue. Implementors of a system where buffer space is not a problem may wish to return without waiting for the buffer to empty whereas implementors of a system with limited buffer space may wish to suspend the operation until the buffer space is emptied. As to which version of asynchronous send operation is being implemented should be transparent and should not affect the correctness of a program. Ambuj Singh Dept. of CS Univ. of California Santa Barbara, CA 93106 ambuj@cs.ucsb.edu From owner-mpi-comm@CS.UTK.EDU Thu Feb 4 19:10:08 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA24655; Thu, 4 Feb 93 19:10:08 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA11923; Thu, 4 Feb 93 19:09:15 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Thu, 4 Feb 1993 19:09:13 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from almaden.ibm.com by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA11915; Thu, 4 Feb 93 19:09:11 -0500 Message-Id: <9302050009.AA11915@CS.UTK.EDU> Received: from almaden.ibm.com by almaden.ibm.com (IBM VM SMTP V2R2) with BSMTP id 9078; Thu, 04 Feb 93 16:09:16 PST Date: Thu, 4 Feb 93 16:09:15 PST From: "Ching-Tien (Howard) Ho" To: mpi-comm@cs.utk.edu Subject: re: > Please remember that overlapped send/receive will only make sense on > systems with excess memory bandwidth. Hence, it might make more sense > to implement a system where this is permitted by the implementor, but > not required. Otherwise, calculations might slow down instead of > achieving pipelined improvements. As Al points out, it is only the > heavy-duty guys who try this sort of stuff... > > - Tony A message passing system with only blocking send and blocking recv is hard to implement a shift or circulant shift efficiently and safely. Either you write an unsafe program bsend (right_nbr) brecv (left_nbr) which may deadlock depending on the size of the system buffer, or you do a two-phase or three-phase algorithm based on parity, which may require pointer jumping and choosing a leader. Regards, -- Howard From owner-mpi-comm@CS.UTK.EDU Thu Feb 4 20:10:38 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA25133; Thu, 4 Feb 93 20:10:38 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA14638; Thu, 4 Feb 93 20:09:30 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Thu, 4 Feb 1993 20:09:29 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from Aurora.CS.MsState.Edu by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA14630; Thu, 4 Feb 93 20:09:28 -0500 Received: by Aurora.CS.MsState.Edu (4.1/6.0s-FWP); id AA00403; Thu, 4 Feb 93 19:09:20 CST Date: Thu, 4 Feb 93 19:09:20 CST From: Tony Skjellum Message-Id: <9302050109.AA00403@Aurora.CS.MsState.Edu> To: ho@almaden.ibm.com Subject: re: my comments on overlapping Cc: mpi-comm@cs.utk.edu BTW, I do not mean blocking send / blocking receive only in my comments. I mean asynchronous reading of a specific message (aka intel irecv) going on at the same time as processing. The message appears in the buffer, and one can poll for that to be complete. In practice, message passing does happen in background on all these systems, but the algorithms are not explicitly using this feature, or encouraging this overlap, necessarily. They might process messages at high priority, and only compute when message input queues are empty. Lennart has always repeated this warning that the overlap is only useful from a performance sense, and therefore from the point of view of a programmer trying to ask for this, if the system can perform better with the overlap. I DO NOT ADVOCATE block send / block receive only. Best wishes, Tony ----- Begin Included Message ----- From owner-mpi-comm@CS.UTK.EDU Thu Feb 4 18:18:36 1993 X-Resent-To: mpi-comm@CS.UTK.EDU ; Thu, 4 Feb 1993 19:09:13 EST Date: Thu, 4 Feb 93 16:09:15 PST From: "Ching-Tien (Howard) Ho" To: mpi-comm@cs.utk.edu Subject: re: Content-Length: 831 > Please remember that overlapped send/receive will only make sense on > systems with excess memory bandwidth. Hence, it might make more sense > to implement a system where this is permitted by the implementor, but > not required. Otherwise, calculations might slow down instead of > achieving pipelined improvements. As Al points out, it is only the > heavy-duty guys who try this sort of stuff... > > - Tony A message passing system with only blocking send and blocking recv is hard to implement a shift or circulant shift efficiently and safely. Either you write an unsafe program bsend (right_nbr) brecv (left_nbr) which may deadlock depending on the size of the system buffer, or you do a two-phase or three-phase algorithm based on parity, which may require pointer jumping and choosing a leader. Regards, -- Howard ----- End Included Message ----- From owner-mpi-comm@CS.UTK.EDU Fri Feb 5 18:05:15 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA14845; Fri, 5 Feb 93 18:05:15 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA03769; Fri, 5 Feb 93 18:03:08 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Fri, 5 Feb 1993 18:03:06 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from ssdintel.ssd.intel.com by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA03760; Fri, 5 Feb 93 18:03:04 -0500 Received: from fuji.ssd.intel.com by SSD.intel.com (4.1/SMI-4.1) id AA06572; Fri, 5 Feb 93 15:02:57 PST Message-Id: <9302052302.AA06572@SSD.intel.com> To: jwf@lion.parasoft.com (Jon Flower) Cc: mpi-comm@cs.utk.edu, prp@SSD.intel.com Subject: Re: 5184; one small point In-Reply-To: Your message of "Thu, 04 Feb 93 14:01:00 PST." Date: Fri, 05 Feb 93 15:02:57 -0800 From: prp@SSD.intel.com Folks, here is an offline exchange Jon and I had that you might enjoy. To: prp@SSD.intel.com Subject: Re: 5184; one small point Paul, I saw your message about blocking and synchronous messages. Without wanting to speak for Chuck I think there is a pretty important difference between the two, particularly for applications such as Oracle. Using the blocking send it is true that the send buffer can be reused as soon as it is "clear" AS LONG AS IT IS GUARANTEED that it will be delivered. Unfortuntely this can't be guaranteed in the case where system buffers are required because the system may simply be out of memory at the receiver. In this case MPI and other systems will simply drop the offending message on the ground. For a database I assume this would be fatal. On the other hand the synchronous scheme has the advantage that you know it's worked when it returns, on both send and receive sides. Jon Flower This is an area where we must be clear about the semantics. We have agreed to some extent that message passing in MPI will be reliable. If I remember correctly, this passed on a straw vote early in the first point-to-point discussion. When I say "agreed to some extent" I mean that not everyone has the same understanding of what the standard is about on this issue. To me it means that if I have a send, and a receive that matches the message sent, that the message will absolutely be delivered. This is how NX works. If there isn't enough buffer space, the send must block so that the message waits in the send buffer until it can be delivered. Flow control is required in the underlying protocol. There seem to be a lot of people that expect no underlying protocol. When you say "send", the system sends - whether the message can be delivered or not. Apparently some systems work this way, and therefore it is expected behavior for people who have been users of these systems. To me, it is not at all compatible with the notion of reliable message passing. If you have reliable message passing as I understand it, clearly the important difference you mention goes away, leaving the lesser difference I wrote about. Without reliable message passing, I see no point whatsoever in blocking send. In fact, I see little point in having a typed send/receive standard, because such an interface is loaded up with friendly help for the user and it seems ridiculous to turn around and expect that same user to worry about reliable delivery. Does any of this make sense? Paul Hi Paul. I think what you're saying makes some sense. I have two more questions though.... a) Having the sender block until memory is available potentially invalidates some algorithms, particularly as the number of nodes grows. It's quite easy to come up with a sort of "ring" algorithm in which everyone ends up hung. I would agree that the trivial case (a single one-dimensional ring) in which this happens is a "user error" but there are more complex multi-dimensional cases in which it is quite hard for users to guarantee that it won't come up, even with quite reasonable programs. b) What are the implications of the flow control that you are using on performance? Is your routing hardware doing the claiming of memory blocks on the receiver or is there an initial software handshake to get started. The latter probably costs a fair amount that the "high priests" don't like??? I agree that the definition of "reliable devliery" in the MPI picture is not solid. I thought I remembered a piece that actually said that it was OK to drop a message if there wasn't enough memory on the receiver but I could be hallucinating. I wonder if it's worth trying to straighten this out? I have a nasty feeling that we'll get down to implementation details. If we can't get IBM and TMC to agree that Node 0: send to 1 read from 1 Node 1: send to 0 read from 0 is a reasonable thing to do I don't see much hope for anything else at this level....... Jon > a) Having the sender block until memory is available... This is inherently a hard problem that is all about system buffering. You can't solve this problem by careful or clever definition of the semantics of a message passing interface (whether and when sends block, and whether message passing is reliable.) If you push it down here, it pops up over there - depending on the semantics defined for the interface, the problem must be solved by the system or by the user. My preference (for a typed send/receive interface) is to push the problem to the system side, as much as possible. That way most users won't have to deal with it. But you can't completely solve the problem on the system side, because the user can always write a program that will confound any system buffering that has finite buffer space. With a reasonably good system implementation, it is actually rather difficult to come up with an algorithm that breaks it. Also, it is possible to define straightforward rules that will guarantee correct execution. That way, although the user must be careful being careful is easy. > b) What are the implications of the flow control... There is a performance penalty. People have claimed that NX overheads are too high, and it is true that they are noticeably higher than raw hardware. But there is quite a bit of overhead from message matching too, so for those who want ultimate performance there should be an alternative to the typed send/receive interface. Active messages is a good candidate. Such an interface is outside the scope of MPI. > I agree that the definition of "reliable devliery" in the MPI > picture is not solid. I thought I remembered a piece that actually > said that it was OK to drop a message if there wasn't enough > memory on the receiver but I could be hallucinating. I wonder It was in the first draft. > if it's worth trying to straighten this out? I have a nasty > feeling that we'll get down to implementation details. If we > can't get IBM and TMC to agree that > > Node 0: > send to 1 > read from 1 > > Node 1: > send to 0 > read from 0 > > is a reasonable thing to do I don't see much hope for anything > else at this level....... We have to develop and agree on a way to manage system buffering. To get Intel, IBM, and TMC together it must allow for the case where the above works and the case where it doesn't. > > Jon Paul Do you have any plan for how to reconcile the difference between the three camps on the simple program I sent you? My impression is that: NX (& Express, in fact) want that program to work as well as it possibly can. I believe that >95% of Express users code that way and are successful. I think you mentioned a similar fact in Dallas. TMC is prepared to say that the program is incorrect and won't work at all. IBM is not convinced either way. I don't see how to reconcile these camps in any good way. If we allow for the possibility that the code doesn't work AT ALL then the 95% of Express codes I mentioned above ALL have to be re-written in order to become portable. Furthermore, if the only way to guarantee portability, even within MPI, is to assume "half-duplex" communication then why not toss the other kind completely -- it would certainly reduce the complexity of the draft? Jon I think the answer (if there is one) is to provide a mechanism where the user declares a need for system buffering. This is a job for the system environment group to work out properly. With a declaration, you get: - Portability: The code is written for a specific buffering model but will port between environments because it tells the system what it needs. - Efficiency: If the user declares no buffering, the system can eliminate the associated overhead. (How to do this is a trick.) - Bloat: Every vendor must provide system buffering in case the user declares a need for it. Paul I agree....... do you think TMC will? Should we attempt to make a formal suggestion along these lines to the entire group. "Prepare to be flamed!!!!" Jon [I'm ready] Paul From owner-mpi-comm@CS.UTK.EDU Fri Feb 5 18:59:06 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA15545; Fri, 5 Feb 93 18:59:06 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA06448; Fri, 5 Feb 93 18:57:54 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Fri, 5 Feb 1993 18:57:52 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from hub.ucsb.edu by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA06440; Fri, 5 Feb 93 18:57:50 -0500 Received: from garuda (garuda.ucsb.edu) by hub.ucsb.edu; id AA18688 sendmail 4.1/UCSB-2.0-sun Fri, 5 Feb 93 15:55:42 PST for mpi-comm@cs.utk.edu Received: by garuda (4.1/UCSB-v2) id AA23498; Fri, 5 Feb 93 14:42:13 PST Date: Fri, 5 Feb 93 14:42:13 PST From: ambuj%cs@hub.ucsb.edu (Ambuj Singh) Message-Id: <9302052242.AA23498@garuda> To: mpi-comm@cs.utk.edu Subject: order preservation Hello: The issue of order preservation of messages when there are multiple threads at the sender was not clarified at the January meeting. What do people have to say about this? It seems that there is no need to require message orderings across different threads. In other words, the order that the messages are received should be consistent with the ``program order'' which is defined by the order in which statements get executed. Thus messages in different threads may not be related and there should not be a need to preserve their order. On a different note, there was some question regarding the need for a synchronous send operation. One reason that they may be useful is that they might be the only means to ensure true portability. As illustrated by the following program that we discussed at the January meeting, asynchronous send operations are not portable across CM-5 and Intel. Process 1 Process 2 --------- --------- send to 2 send to 1 recv recv In other words, the standard could inquire that programs employing synchronous send/receive operations be portable across machines. It would be difficult to meet the same requirement for the asynchronous operations. --Ambuj Singh Dept. of CS, UCSB From owner-mpi-comm@CS.UTK.EDU Mon Feb 8 06:54:25 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA18563; Mon, 8 Feb 93 06:54:25 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA17241; Mon, 8 Feb 93 06:53:03 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Mon, 8 Feb 1993 06:53:02 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from marge.meiko.com by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA17231; Mon, 8 Feb 93 06:52:56 -0500 Received: from hub.meiko.co.uk by marge.meiko.com with SMTP id AA03743 (5.65c/IDA-1.4.4 for ); Mon, 8 Feb 1993 06:52:53 -0500 Received: from float.co.uk (float.meiko.co.uk) by hub.meiko.co.uk (4.1/SMI-4.1) id AA26581; Mon, 8 Feb 93 11:52:48 GMT Date: Mon, 8 Feb 93 11:52:48 GMT From: jim@meiko.co.uk (James Cownie) Message-Id: <9302081152.AA26581@hub.meiko.co.uk> Received: by float.co.uk (5.0/SMI-SVR4) id AA00721; Mon, 8 Feb 93 11:50:58 GMT To: mpi-comm@cs.utk.edu In-Reply-To: prp@SSD.intel.com's message of Fri, 05 Feb 93 15:02:57 -0800 <9302052302.AA06572@SSD.intel.com> Subject: The Jon and Paul show Content-Length: 1971 While I am no fan of arbitrary system buffering of messages, I believe that if it is to succeed MPI must allow the easy importation of existing Express and NX style codes. This necessarily implies that system buffering is required, so that code like this Blocking send to myself Blocking receive from myself works. (This is an even simpler version of the test case given by Jon/Paul). I agree with the approach that the user should declare her buffering requirement up front. This seems to be the only way to ensure portability. (Even if all it achieves is ensuring that she gets an "MPI-INSFMEM insufficient buffer space available" message on some systems, this is still preferable to a crash or hang). This is exactly the set of issues Marc was addressing in his discussion on SAFE and UNSAFE programs, and the amounts of buffering such codes could assume (he also addressed issues such as "How many outstanding messages can a code assume it is allowed ?"). I've a feeling that a lot of users (most ?) don't actually know what their buffer requirement is, however I still think that requiring a user specification of required buffer space is the correct way forward. (It might also be nice to have a system enquiry function which could return the buffer space currently used and the high water mark. This would at least let users find out about the behaviour of their codes. Unfortunately this information is rather implementation specific, and would not be accessible through the currently proposed profiling interface. Is this an issue for the profiling committee, or the environmental enquiry one ??? [As chair of the Profiling committe I'd vote for Environmental Enquiry !]) -- Jim James Cownie Meiko Limited Meiko Inc. 650 Aztec West Reservoir Place Bristol BS12 4SD 1601 Trapelo Road England Waltham MA 02154 Phone : +44 454 616171 +1 617 890 7676 FAX : +44 454 618188 +1 617 890 5042 E-Mail: jim@meiko.co.uk or jim@meiko.com From owner-mpi-comm@CS.UTK.EDU Mon Feb 8 07:10:19 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA22055; Mon, 8 Feb 93 07:10:19 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA17649; Mon, 8 Feb 93 07:09:28 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Mon, 8 Feb 1993 07:09:27 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from marge.meiko.com by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA17641; Mon, 8 Feb 93 07:09:23 -0500 Received: from hub.meiko.co.uk by marge.meiko.com with SMTP id AA03842 (5.65c/IDA-1.4.4 for ); Mon, 8 Feb 1993 07:09:20 -0500 Received: from float.co.uk (float.meiko.co.uk) by hub.meiko.co.uk (4.1/SMI-4.1) id AA26809; Mon, 8 Feb 93 12:09:13 GMT Date: Mon, 8 Feb 93 12:09:13 GMT From: jim@meiko.co.uk (James Cownie) Message-Id: <9302081209.AA26809@hub.meiko.co.uk> Received: by float.co.uk (5.0/SMI-SVR4) id AA00724; Mon, 8 Feb 93 12:07:22 GMT To: mpi-comm@cs.utk.edu In-Reply-To: prp@SSD.intel.com's message of Fri, 05 Feb 93 15:02:57 -0800 <9302052302.AA06572@SSD.intel.com> Subject: The Jon and Paul show [more] Content-Length: 2029 One more point I ommitted before... Paul says :- > to work out properly. With a declaration, you get: > > - Portability: The code is written for a specific buffering model but will > port between environments because it tells the system what it needs. > > - Efficiency: If the user declares no buffering, the system can eliminate > the associated overhead. (How to do this is a trick.) > > - Bloat: Every vendor must provide system buffering in case the user declares > a need for it. The last point is not actually true. It's entirely reasonable to have a standard which contains a buffering request and allow it to be refused. (There's not actually any point in having the request unless it can be refused !) I don't see a huge logical distinction between an implementation which permits system buffering of 1 byte and one which permits no system buffering. If the user needed 1Mb, then neither system is any use ! So it's merely an implementation quality issue. If someone has been able to sell their current system and keep their users happy without system buffering, then they should still be able to keep them happy with an MPI implementation in which the amount of system buffering is zero, and any request for more fails. Such an implementation would be standard conforming too. (Though such a vendor might expect some moaning from people trying to import code onto the machine, and maybe market pressure would push them into an implementation which did support system buffering !) (As an analogy, I believe that it is possible to have a standard conforming ADA implementation whose sole action when presented with any program is to raise the "Implementation limit exceeded" exception. Not useful, not saleable, but standard !) -- Jim James Cownie Meiko Limited Meiko Inc. 650 Aztec West Reservoir Place Bristol BS12 4SD 1601 Trapelo Road England Waltham MA 02154 Phone : +44 454 616171 +1 617 890 7676 FAX : +44 454 618188 +1 617 890 5042 E-Mail: jim@meiko.co.uk or jim@meiko.com From owner-mpi-comm@CS.UTK.EDU Mon Feb 8 08:54:49 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA26134; Mon, 8 Feb 93 08:54:49 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA20868; Mon, 8 Feb 93 08:54:03 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Mon, 8 Feb 1993 08:54:02 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from [129.215.56.21] by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA20822; Mon, 8 Feb 93 08:52:39 -0500 Date: Mon, 8 Feb 93 13:52:17 GMT Message-Id: <16939.9302081352@subnode.epcc.ed.ac.uk> From: L J Clarke Subject: buffered/unbuffered comms (was compiler target (was Be brave. Be sure.)) To: mpi-pt2pt@cs.utk.edu, mpi-comm@cs.utk.edu Reply-To: lyndon@epcc.ed.ac.uk Hi Here comes my 5p worth on the subject of buffered and unbuffered comms. For completeness: by unbuffered I mean that the sender blocks until the message has been (or certainly will be) copied into the space of the receiver, a la occam for example; by buffered comms I mean that the message is copied away into a system buffer somewhere and the sender returns, a la PVM for example. It seems to me that there are a few different issues in this subject which discussions may have touched on. a) Ease of programming. I don't think anyone can really disagree that many programmers will report that they find buffered comms easier to use. These users have been fortunate enough not to have come up against the boundedness of buffering provided with the system. With high probability they are programming (or adapting) applications by inserting message passing calls directly into the application source. b) Portability of programs using MPI between different machines. [Here I give away my bias.] It is very difficult to make statements about portability (and reliability/correctness) of programs which use buffered messages and rely on system storage capacity and structure thereof. This becomes particularly difficult when the application makes use of substantive libraries, which themselves are written using buffered comms and place requirements on system buffering. It can just become to difficult to work out how much buffering you need. We played with a system which allowed the programmer to configure the system buffering, and this feature was only used (properly) by high priests. In my work I will only be able to use MPI if I can write substantive libraries in the confidence that they will not be subject to failure due to running out of such buffer space. Therefore we use only unbuffered message passing with a mixture of blocking/nonblocking (like Intel NX isend/irecv/msgwait) calls. Therefore my bias :-) c) Portability of existing applications (using existing message passing interfaces) to MPI. This is a different subject to (c). Following the discussion it seems that an argument in favour of adopting buffered comms is the number of existing programs (eg, lotsa programs written using Express) which use unbuffered comms. It has been my experience that migration of applications between different, broadly similar, message passing interfaces is not so difficult, except for this point. In a nutshell, the surgery you have to perform to move programs/libraries between an interface that forces buffered comms and one which forces unbuffered comms and blocking/nonblocking (isend/irecv/msgwait) is grevious and error prone. Given the above, I come to the conclusions: i) MPI must contain unbuffered communications with blocking/nonblocking (irecv/isend/msgwait kinda thing) calls, for reliability and portability. ii) If a goal of MPI is that existing applications using message passing interfaces (eg Express, PVM) should easily port to MPI, then MPI must also contain buffered comms. This seems to be a matter for the full committee, hence I have crossposted. Incidentally, to pitch in 2ps on some other lines of discussion: * I can see no particular benefit in allowing a buffered snd/rcv match an unbuffered rcv/snd. I can see considerable inconvenience in forbidding a blocking send/recv to match with a nonblocking recv/send. * Yes, we do try to overlap communications with calculation. (Sometimes it doesnt buy you, sometimes it does. Sometimes we have specifically needed to avoid such overlap in order to maximise performance, but that was a weird one :-) Just as important, we frequently "overlap" communication with communication (ie, use nonblocking calls) to avoid deadlock. * Please, oh please, let MPI decide that communications should be adressed using a rank relative to a group/context (0 ... GroupSize - 1) or (1 ... GroupSize). We do this all the time, and its very very convenient. In fact, when we can't do this we end up having to create arrays of the task identifiers - we end up doing it ourselves anyway. Best Wishes Lyndon /--------------------------------------------------------\ e||) | Lyndon J Clarke Edinburgh Parallel Computing Centre | e||) c||c | Tel: 031 650 5021 Email: lyndon@epcc.edinburgh.ac.uk | c||c \--------------------------------------------------------/ From owner-mpi-comm@CS.UTK.EDU Tue Feb 9 02:23:46 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA19163; Tue, 9 Feb 93 02:23:46 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA06697; Tue, 9 Feb 93 02:22:48 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Tue, 9 Feb 1993 02:22:47 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from antares.mcs.anl.gov by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA06689; Tue, 9 Feb 93 02:22:45 -0500 Received: from donner.mcs.anl.gov by antares.mcs.anl.gov (4.1/SMI-GAR) id AA20792; Tue, 9 Feb 93 01:22:44 CST Received: by donner.mcs.anl.gov (4.1/GCF-5.8) id AA04771; Tue, 9 Feb 93 01:22:41 CST Message-Id: <9302090722.AA04771@donner.mcs.anl.gov> To: mpi-comm@cs.utk.edu Subject: The Jon and Paul show (was: 5184; one small point) In-Reply-To: Your message of "Fri, 05 Feb 93 15:02:57 PST." <9302052302.AA06572@SSD.intel.com> Date: Tue, 09 Feb 93 01:22:40 CST From: Rusty Lusk | | Folks, here is an offline exchange Jon and I had that you might enjoy. .... | Should we attempt to make a formal suggestion along these lines | to the entire group. "Prepare to be flamed!!!!" (not a flame) I did enjoy this exchange, because it is a tutorial on the buffering problem. Paul says it perfectly: You can't solve the system buffering problem by careful or clever definition of the semantics of a message passing interface. I agree that most users would rather not worry about it (and don't) and we should allow systems that try to handle the problem for the user to do so (the example programs in the discussion should run). I agree with Jim Cownie on this point. For portability, an enquiry function can be provided to alert the user who checks to find out how much (or how little) buffering is provided. I think the environmental enquiry subcommittee should propose something along these lines. As Paul says, handling this robustly has a cost, and some users certainly want to avoid it. After all, the user can know some things about his program that the system cannot, and should be able to take over responsibility for the buffering problem if he wants to. This is just what is done in the case of the "ready-receiver" communication. I admit that this is not a particularly euphonious name, but some way is needed for the user to relieve the system of buffer management if he wants to do so in the interest of greater efficiency. Rusty From owner-mpi-comm@CS.UTK.EDU Wed Feb 10 01:16:47 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA13938; Wed, 10 Feb 93 01:16:47 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA11091; Wed, 10 Feb 93 01:15:37 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Wed, 10 Feb 1993 01:15:35 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from antares.mcs.anl.gov by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA11083; Wed, 10 Feb 93 01:15:33 -0500 Received: from donner.mcs.anl.gov by antares.mcs.anl.gov (4.1/SMI-GAR) id AA02367; Wed, 10 Feb 93 00:15:31 CST Received: by donner.mcs.anl.gov (4.1/GCF-5.8) id AA06777; Wed, 10 Feb 93 00:15:28 CST Message-Id: <9302100615.AA06777@donner.mcs.anl.gov> To: mpi-comm@cs.utk.edu Subject: Multiple levels of collective operations Date: Wed, 10 Feb 93 00:15:27 CST From: Rusty Lusk A few days ago, we posted a suggestion for a multi-level MPI specification as a way to manage the complexity inherent in providing a highly functional interface, and to make MPI accessible to users who did not need or want all of its capabilities. The examples there were only for point-to-point communication. We thought it worth exploring whether the same ideas might be useful in the context of the group operations. Here are a few thoughts. The idea here is not to propose specific syntax, but to suggest how simultaneous adoption of multiple levels of collective operations might help provide both a simple interface and a full-featured one. Level 4: (very, very simple) Here there would be only one group, the default group of all processes, so there is no group id in the parameter lists. MPI_barrier MPI_combine(...,SUM,...) MAX ... (That is, only a fixed set of combining operations) MPI_broadcast(....) All operations would block until globally completed. Message types would be just like level 4 of the point-to-point operations: either contiguous strings of bytes, or arrays of specific data types specified like (...,datatype,numitems) The main point is that using just this level of group operations one could port many existing programs that use global operations, without introducing any explicit management of groups. Level 3: (still pretty simple, but more functionality) All the utilities proposed by Al Geist at the last meeting for the management of groups (creation, inquiry, etc.) Operations at this level have group id as an explicit argument (Again as Al suggested). User-defined combining functions. Gather operations. Main restriction: for simplicity, still simple contiguous buffers made up of either untyped byte strings or arrays of fixed types. All operations still block. In fact, we propose that there not be non-blocking collective operations, even at lower levels. The necessity of cooperating in the global operation beyond just making one's own explicit contribution puts non-blocking global operations more in the "threads" domain. Level 2: (More options for buffer structure, as in the point-to-point routines at this level) Here buffers specified for broadcasting, combining, etc., could be completely general, including mixed-type, non-contiguous buffers described by vectors of (address,datatype,numitems) triples. Level 1: (Full functionality, plus separate "setup" parts of the operations, like in Marc's point-to-point proposal.) Here operations would have a separate "init" call. Being separate has the same advantages as for the point-to-point operations, in that resources could be reused in loops. The "init" call would return a handle that could be further modified (setting buffer type, address, etc.) before use. The payoff seems potentially greater here because some of the "init" calls may turn out to be inherently non-scalable, and therefore reuse of their results is important. Summary: Multiple levels might help deal with conflicting demands for both simplicity and functionality in the case of collective operations as well as for point-to-point operations. What was generally discussed at the last meeting is very much like Level 3 above. An even simpler interface (so that groups need not be explicit at all) is provided by Level 4, while the somewhat grubby aspects of multiple and complicated buffer formats is pushed down into Level 2. Finally, Level 1 provides the opportunity to absorb some of the difficult-to-scale parts of global operations into separate, reusable calls. Comments? Rusty Lusk and Bill Gropp From owner-mpi-comm@CS.UTK.EDU Fri Feb 12 04:14:26 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA05929; Fri, 12 Feb 93 04:14:26 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA23402; Fri, 12 Feb 93 04:12:57 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Fri, 12 Feb 1993 04:12:57 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from enseeiht.enseeiht.fr by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA23393; Fri, 12 Feb 93 04:12:50 -0500 Received: from marylin (marylin.enseeiht.fr) by enseeiht.enseeiht.fr, Fri, 12 Feb 1993 10:12:45 +0100 Message-Id: <199302120912.AA08776@enseeiht.enseeiht.fr> Date: Fri, 12 Feb 93 10:10:29 +0100 From: Michel DAYDE To: mpi-comm@cs.utk.edu Subject: Re Multiple levels of cllective operations I like the idea of multi-level MPI specifications since it is a nice way of dealing with functionality and simplicity. Also the fact that Level 4 of collective operations avoids to introduce any explicit management of groups will simplify the life. Michel Dayde, ENSEEIHT-IRIT and CERFACS. From owner-mpi-comm@CS.UTK.EDU Mon Feb 15 07:56:19 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA02001; Mon, 15 Feb 93 07:56:19 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA14309; Mon, 15 Feb 93 07:54:13 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Mon, 15 Feb 1993 07:54:12 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from daedalus.epcc.ed.ac.uk by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA14301; Mon, 15 Feb 93 07:54:07 -0500 Date: Mon, 15 Feb 93 12:54:01 GMT Message-Id: <21675.9302151254@subnode.epcc.ed.ac.uk> From: L J Clarke Subject: Mixed F77/C, MIMD(not SPMD) To: mpi-comm@cs.utk.edu Reply-To: lyndon@epcc.ed.ac.uk Dear MPI Colleagues I'd like to raise a couple of points which I hope we can discuss at the meeting later this week, after lenghty discussions with a variety of local users. In general the objectives of the MPI Forum are very well received. Two specific concerns, of a very general nature, became apparent. a) The introduction says that MPI is for programmers who wish to write portable programs in C and/or Fortran77. The choice of languages is fine, although some may wish to use C++ and Fortran90. The concern is whether MPI intends to allow, and if so in what manner, mixed language programming, i.e. programs written in a mixture of C and Fortran77. The observation is that C is most useful for library programming, whereas a large number of applications that we deal with are written in Fortran77. (Users did not expect MPI say anything about cross-calling and such murky business, that is outwith message passing.) I propose that the make a statement of intent, possibly in the introduction, that mixed languages will be allowed/supported (I would hope that we could not make the decision to not allow/support mixture of C and F77 :-). This seems an easy thing to do. b) The working introduction of November 24 states that MPI includes (temporarily?) \item A simple way to create processes for the SPMD model This kinda carries the implication that MPI is for programming in the SPMD model. The January minutes state that It was agreed that explicit remarks that MPI is intended to be usable with multithreaded proesses and with MIMD (not just SPMD) programs should be added somewhere. We at EPCC most strongly support the proposal that MPI should provide for MIMD programs and not restrict itself to the world of SPMD programming. The SPMD model has been useful, and can be expected to remain useful, for tweaking serial (usually Fortran77) programs in a strictly data parallel fashion to derive a parallel program. However, this is not a string enough argument for restricting MPI to SPMD. There are a lot of applications out there which are not really suited to such simple tweaking. Further, we believe that MPI needs to provide support for library writers, where libraries can consist of a procedure library linked into user processes along with one or more library service processes. The concern felt by users and gurus alike is that our perception of much of the discussion in subcommittees is that it seems to be proceeding with scant consideration of issues in MIMD programming as opposed to SPMD programming. I suggest that this should be prioritised within the MPI. I am happy to write a proposal or discussion document on these issues, but I have no time to write either before the next meeting as my travel schedule starts at 6am tomorrow. So, I'm afraid this point was simply raising a flag that there is concern, without a proposal on the matter, for which I apologise. Looking forward to meeting and discussing at the meeting later this week! Best Wishes Lyndon /--------------------------------------------------------\ e||) | Lyndon J Clarke Edinburgh Parallel Computing Centre | e||) c||c | Tel: 031 650 5021 Email: lyndon@epcc.edinburgh.ac.uk | c||c \--------------------------------------------------------/ From owner-mpi-comm@CS.UTK.EDU Mon Feb 15 10:21:06 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA04616; Mon, 15 Feb 93 10:21:06 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA20711; Mon, 15 Feb 93 10:19:40 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Mon, 15 Feb 1993 10:19:38 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from antares.mcs.anl.gov by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA20702; Mon, 15 Feb 93 10:19:36 -0500 Received: from godzilla.mcs.anl.gov by antares.mcs.anl.gov (4.1/SMI-GAR) id AA15992; Mon, 15 Feb 93 09:19:33 CST From: gropp@antares.mcs.anl.gov (William Gropp) Received: by godzilla.mcs.anl.gov (4.1/GeneV4) id AA27181; Mon, 15 Feb 93 09:19:31 CST Date: Mon, 15 Feb 93 09:19:31 CST Message-Id: <9302151519.AA27181@godzilla.mcs.anl.gov> To: lyndon@epcc.ed.ac.uk Cc: mpi-comm@cs.utk.edu In-Reply-To: L J Clarke's message of Mon, 15 Feb 93 12:54:01 GMT <21675.9302151254@subnode.epcc.ed.ac.uk> Subject: Mixed F77/C, MIMD(not SPMD) X-Resent-To: mpi-comm@CS.UTK.EDU ; Mon, 15 Feb 1993 07:54:12 EST Date: Mon, 15 Feb 93 12:54:01 GMT From: L J Clarke Reply-To: lyndon@epcc.ed.ac.uk Again, a few clarifications: b) The working introduction of November 24 states that MPI includes (temporarily?) \item A simple way to create processes for the SPMD model This kinda carries the implication that MPI is for programming in the SPMD model. My interpretation (and in fact, the reason that Rusty and I included such a method in our implementation of the November draft) is that there needs to be some way to write, entirely in MPI, a running program, and there is a chance that we could agree on a method for running SPMD programs. It was definately NOT meant to exclude other models; just a position that MPI needs to include some way to write a complete (at the source code level; we do not intend to specify the OS interface to "launching" the program) parallel program for a common and simple model. I'd be quite happy adding text that indicates that MPI is intended for MIMD models as well. I would be very interested in seeing a proposal for an interface for MIMD programs. Bill From owner-mpi-comm@CS.UTK.EDU Mon Feb 15 10:21:06 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA04617; Mon, 15 Feb 93 10:21:06 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA20721; Mon, 15 Feb 93 10:19:42 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Mon, 15 Feb 1993 10:19:41 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from antares.mcs.anl.gov by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA20713; Mon, 15 Feb 93 10:19:40 -0500 Received: from jadoube.mcs.anl.gov by antares.mcs.anl.gov (4.1/SMI-GAR) id AA15995; Mon, 15 Feb 93 09:19:37 CST From: levine@antares.mcs.anl.gov (David Levine) Received: by jadoube.mcs.anl.gov (4.1/GeneV4) id AA21322; Mon, 15 Feb 93 09:19:36 CST Date: Mon, 15 Feb 93 09:19:36 CST Message-Id: <9302151519.AA21322@jadoube.mcs.anl.gov> To: lyndon@epcc.ed.ac.uk Cc: mpi-comm@cs.utk.edu Subject: Mixed F77/C, MIMD(not SPMD) X-Resent-To: mpi-comm@CS.UTK.EDU ; Mon, 15 Feb 1993 07:54:12 EST Date: Mon, 15 Feb 93 12:54:01 GMT From: L J Clarke Reply-To: lyndon@epcc.ed.ac.uk Dear MPI Colleagues I'd like to raise a couple of points which I hope we can discuss at the meeting later this week, after lenghty discussions with a variety of local users. In general the objectives of the MPI Forum are very well received. Two specific concerns, of a very general nature, became apparent. a) The introduction says that MPI is for programmers who wish to write portable programs in C and/or Fortran77. The choice of languages is fine, although some may wish to use C++ and Fortran90. The concern is whether MPI intends to allow, and if so in what manner, mixed language programming, i.e. programs written in a mixture of C and Fortran77. The observation is that C is most useful for library programming, whereas a large number of applications that we deal with are written in Fortran77. (Users did not expect MPI say anything about cross-calling and such murky business, that is outwith message passing.) I propose that the make a statement of intent, possibly in the introduction, that mixed languages will be allowed/supported (I would hope that we could not make the decision to not allow/support mixture of C and F77 :-). This seems an easy thing to do. Let me second this, if for no other reason than the lack of dynamic memory allocation in Fortran 77, and the relatively small physical memory available on the nodes of distributed-memory machines. In porting Fortran 77 codes to distributed-memory machines I consider the use of C for dynamic memory allocation to be "mandatory." --dave levine From owner-mpi-comm@CS.UTK.EDU Mon Feb 15 12:20:23 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA07231; Mon, 15 Feb 93 12:20:23 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA25990; Mon, 15 Feb 93 12:18:19 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Mon, 15 Feb 1993 12:18:17 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from chert.CS.ORST.EDU by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA25979; Mon, 15 Feb 93 12:18:15 -0500 Received: from sycamore.CS.ORST.EDU by research.CS.ORST.EDU (4.1/1.30) id AA03629; Mon, 15 Feb 93 09:18:12 PST Date: Mon, 15 Feb 93 09:18:12 PST From: pancake@chert.CS.ORST.EDU (Cherri Pancake) Message-Id: <9302151718.AA03629@research.CS.ORST.EDU> To: mpi-comm@cs.utk.edu Subject: C/Fortran Compatibility I agree with Lyndon Clarke that interlanguage compatibility needs to be an explicitly stated goal of the standard. The user community is assuming that this is true.... Cherri Pancake From owner-mpi-comm@CS.UTK.EDU Mon Feb 15 13:17:38 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA09917; Mon, 15 Feb 93 13:17:38 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA29053; Mon, 15 Feb 93 13:16:12 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Mon, 15 Feb 1993 13:16:11 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from NA-GW.CS.YALE.EDU by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA29045; Mon, 15 Feb 93 13:16:09 -0500 Received: from YOGI.NA.CS.YALE.EDU by CASPER.NA.CS.YALE.EDU via SMTP; Mon, 15 Feb 1993 13:16:06 -0500 Received: by YOGI.NA.CS.YALE.EDU (Sendmail-5.65c/res.client.cf-3.5) id AA28124; Mon, 15 Feb 1993 13:16:04 -0500 Date: Mon, 15 Feb 1993 13:16:04 -0500 From: berryman-harry@CS.YALE.EDU (Harry Berryman) Message-Id: <199302151816.AA28124@YOGI.NA.CS.YALE.EDU> To: mpi-comm@cs.utk.edu Subject: C/Fortran Compatibility Ok, folks, What exactly do you mean by "interlanguage compatibility"? In particular are we going to ask that F77 programs be able to call C routines? Although I agree that this is a good idea, I don't think that dictating compiler (and linker) standards is in the jurisdiction of this committee. On a slightly different issue, it is not clear to me that a C++ interface would be different from an ANSI C interface. Although it is possible to make an object-oriented interface definition, I think that such a definition would be unwise for several reasons. First, if an ANSI C interface is available, anyone wanting an object oriented C++ interface could write one in a portable way. Second, I don't think that enough object-oriented message-passing interfaces have been written to determine what a standard should look like. Of course, I'm willing to persuaded on either point. -scott berryman Chairanimal, Language Binding From owner-mpi-comm@CS.UTK.EDU Mon Feb 15 14:00:27 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA11570; Mon, 15 Feb 93 14:00:27 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA01123; Mon, 15 Feb 93 13:59:09 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Mon, 15 Feb 1993 13:59:07 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from Aurora.CS.MsState.Edu by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA01110; Mon, 15 Feb 93 13:59:04 -0500 Received: by Aurora.CS.MsState.Edu (4.1/6.0s-FWP); id AA00836; Mon, 15 Feb 93 12:58:50 CST Date: Mon, 15 Feb 93 12:58:50 CST From: Tony Skjellum Message-Id: <9302151858.AA00836@Aurora.CS.MsState.Edu> To: mpi-comm@cs.utk.edu, berryman-harry@CS.YALE.EDU Subject: Re: C/Fortran Compatibility I agree with Scott. I think that a valid, related point, is that there need to be a C interface which is convenient for C programmers, not just a FORTRAN interface, which is usable by C programmers, but not natural. - Tony ----- Begin Included Message ----- From owner-mpi-comm@CS.UTK.EDU Mon Feb 15 12:41:33 1993 X-Resent-To: mpi-comm@CS.UTK.EDU ; Mon, 15 Feb 1993 13:16:11 EST Date: Mon, 15 Feb 1993 13:16:04 -0500 From: berryman-harry@CS.YALE.EDU (Harry Berryman) To: mpi-comm@cs.utk.edu Subject: C/Fortran Compatibility Content-Length: 937 Ok, folks, What exactly do you mean by "interlanguage compatibility"? In particular are we going to ask that F77 programs be able to call C routines? Although I agree that this is a good idea, I don't think that dictating compiler (and linker) standards is in the jurisdiction of this committee. On a slightly different issue, it is not clear to me that a C++ interface would be different from an ANSI C interface. Although it is possible to make an object-oriented interface definition, I think that such a definition would be unwise for several reasons. First, if an ANSI C interface is available, anyone wanting an object oriented C++ interface could write one in a portable way. Second, I don't think that enough object-oriented message-passing interfaces have been written to determine what a standard should look like. Of course, I'm willing to persuaded on either point. -scott berryman Chairanimal, Language Binding ----- End Included Message ----- From owner-mpi-comm@CS.UTK.EDU Mon Feb 15 14:08:46 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA11680; Mon, 15 Feb 93 14:08:46 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA01543; Mon, 15 Feb 93 14:07:50 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Mon, 15 Feb 1993 14:07:49 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from sol.cs.wmich.edu by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA01534; Mon, 15 Feb 93 14:07:48 -0500 Received: from id.wmich.edu by cs.wmich.edu (4.1/SMI-4.1) id AA01566; Mon, 15 Feb 93 14:00:38 EST Date: Mon, 15 Feb 93 14:00:38 EST From: john@cs.wmich.edu (John Kapenga) Message-Id: <9302151900.AA01566@cs.wmich.edu> To: mpi-comm@CS.UTK.EDU Subject: C/Fortran Compatibility Asking for compatibility is a lot in some respects. going far beyond the MPI interface - such as asking a malloc from a C subroutine to "work" when called from a Fortran program. (library name collisions - need for initializations for both environments, cleaning up both environments etc.) I want to see them together too, but.... Perhaps we should not make it a full "requirement", but rather be careful to set up the MPI language bindings so that they could work together from C and Fortran. and indicate that this is intended on environments supporting mixed programs. I know of no standards work on mixed programs and recall the the discussion of it (ANSI?) as a general area for standardization a couple years back being postponed as too "hairy". cheers - john From owner-mpi-comm@CS.UTK.EDU Mon Feb 15 14:24:08 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA12649; Mon, 15 Feb 93 14:24:08 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA02252; Mon, 15 Feb 93 14:23:11 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Mon, 15 Feb 1993 14:23:10 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from chert.CS.ORST.EDU by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA02243; Mon, 15 Feb 93 14:23:08 -0500 Received: from sycamore.CS.ORST.EDU by research.CS.ORST.EDU (4.1/1.30) id AA06780; Mon, 15 Feb 93 11:23:02 PST Date: Mon, 15 Feb 93 11:23:02 PST From: pancake@chert.CS.ORST.EDU (Cherri Pancake) Message-Id: <9302151923.AA06780@research.CS.ORST.EDU> To: mpi-comm@cs.utk.edu Subject: hat is Interlanguage Compatibility? To me, multilanguage support means that: (a) we provide both C and Fortran programmatic interfaces, and (b) operationally, there will be no perceptible difference whether the programmer uses the C interface, the Fortran, or both in his/her program. Cherri Pancake From owner-mpi-comm@CS.UTK.EDU Mon Feb 15 14:36:03 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA13372; Mon, 15 Feb 93 14:36:03 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA02815; Mon, 15 Feb 93 14:34:51 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Mon, 15 Feb 1993 14:34:50 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from NA-GW.CS.YALE.EDU by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA02805; Mon, 15 Feb 93 14:34:49 -0500 Received: from YOGI.NA.CS.YALE.EDU by CASPER.NA.CS.YALE.EDU via SMTP; Mon, 15 Feb 1993 14:34:46 -0500 Received: by YOGI.NA.CS.YALE.EDU (Sendmail-5.65c/res.client.cf-3.5) id AA28237; Mon, 15 Feb 1993 14:34:44 -0500 Date: Mon, 15 Feb 1993 14:34:44 -0500 From: berryman-harry@CS.YALE.EDU (Harry Berryman) Message-Id: <199302151934.AA28237@YOGI.NA.CS.YALE.EDU> To: mpi-comm@cs.utk.edu Subject: Any opinions? In my earlier message, I suggested that a C++ interface was not needed if an ANIS C interface was available. Are there any objections to this? -scott berryman From owner-mpi-comm@CS.UTK.EDU Mon Feb 15 14:44:24 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA13826; Mon, 15 Feb 93 14:44:24 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA03257; Mon, 15 Feb 93 14:43:38 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Mon, 15 Feb 1993 14:43:37 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from NA-GW.CS.YALE.EDU by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA03248; Mon, 15 Feb 93 14:43:36 -0500 Received: from YOGI.NA.CS.YALE.EDU by CASPER.NA.CS.YALE.EDU via SMTP; Mon, 15 Feb 1993 14:43:34 -0500 Received: by YOGI.NA.CS.YALE.EDU (Sendmail-5.65c/res.client.cf-3.5) id AA28255; Mon, 15 Feb 1993 14:43:33 -0500 Date: Mon, 15 Feb 1993 14:43:33 -0500 From: berryman-harry@CS.YALE.EDU (Harry Berryman) Message-Id: <199302151943.AA28255@YOGI.NA.CS.YALE.EDU> To: mpi-comm@cs.utk.edu Subject: Perhaps another place? Cherri, Perhaps we should carry on the thread about C++, ANSI C, F77, and F90 to the mpi-lang mailing list. At any rate, I'm very much for having the interface consistent across languages. This does however, limit what we can do in C++ and F90. We cannot, for example, use methods in C++ or optional arguments in F90. Will such limitations cause much gnashing of teeth in the user community? Will we hate ourselves in a year or two for not taking advantage of the truly rightous F90 features? -scott berryman From owner-mpi-comm@CS.UTK.EDU Mon Feb 15 15:11:20 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA14568; Mon, 15 Feb 93 15:11:20 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA04587; Mon, 15 Feb 93 15:09:52 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Mon, 15 Feb 1993 15:09:51 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from ssdintel.ssd.intel.com by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA04578; Mon, 15 Feb 93 15:09:49 -0500 Received: from tualatin.SSD.intel.com by SSD.intel.com (4.1/SMI-4.1) id AA25287; Mon, 15 Feb 93 11:39:31 PST Date: Mon, 15 Feb 93 11:39:31 PST Message-Id: <9302151939.AA25287@SSD.intel.com> Received: by tualatin.SSD.intel.com (4.1/SMI-4.0) id AA06755; Mon, 15 Feb 93 11:39:30 PST From: Bob Knighten Sender: knighten@SSD.intel.com To: john@cs.wmich.edu Cc: mpi-comm@CS.UTK.EDU Subject: Re: C/Fortran Compatibility In-Reply-To: <9302151900.AA01566@cs.wmich.edu> References: <9302151900.AA01566@cs.wmich.edu> Reply-To: knighten@SSD.intel.com (Bob Knighten) POSIX Standardized Profiles are one standard arena where some effort is being made to guarantee some expected level of interoperability. My POSIX working group (P1003.14) is producing the P1003.18 profile for "classic POSIX" covering P1003.1 (system interfaces) and P1003.2 (shell and utilities). This does not include networks or any of the serious problems regarding interoperability. In this context we have been struggling just to find reasonable requirements to express such expectations as that a file written by a C program can be read by a FORTRAN program or an Ada program. We would not even dream of trying to stadardized the way you might link routines produced by different language compilers! -- Bob Robert L. Knighten | knighten@ssd.intel.com Intel Supercomputer Systems Division | 15201 N.W. Greenbrier Pkwy. | (503) 629-4315 Beaverton, Oregon 97006 | (503) 629-9147 [FAX] From owner-mpi-comm@CS.UTK.EDU Mon Feb 15 17:26:17 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA18114; Mon, 15 Feb 93 17:26:17 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA10980; Mon, 15 Feb 93 17:25:12 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Mon, 15 Feb 1993 17:25:11 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from chert.CS.ORST.EDU by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA10972; Mon, 15 Feb 93 17:25:09 -0500 Received: from sycamore.CS.ORST.EDU by research.CS.ORST.EDU (4.1/1.30) id AA11917; Mon, 15 Feb 93 14:25:03 PST From: pancake@chert.CS.ORST.EDU (Cherri Pancake) Received: by sycamore.CS.ORST.EDU (4.1/CS-Client) id AA10017; Mon, 15 Feb 93 14:24:59 PST Date: Mon, 15 Feb 93 14:24:59 PST Message-Id: <9302152224.AA10017@sycamore.CS.ORST.EDU> To: mpi-comm@cs.utk.edu Subject: Response to Bob Knighten's comments Bob said: ?We would not even dream of trying to >stadardized the way you might link routines produced by different language >compilers! I don't think that's necessarily the goal of multilanguage support. I should be able to write a program that combines routines written in Fortran and C, using MPI for the message-passing. That does not mean that the function calls look identical in the two source languages, nor that there be a single set of library routines capable of servicing both. But it does mean that I should be able to send a message from a Fortran routine, calling the Fortran library module, and receive it from a C routine that uses the corresponding C library module. Cherri Pancake From owner-mpi-comm@CS.UTK.EDU Mon Feb 15 17:45:07 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA19007; Mon, 15 Feb 93 17:45:07 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA11672; Mon, 15 Feb 93 17:43:36 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Mon, 15 Feb 1993 17:43:35 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from enet-gw.pa.dec.com by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA11664; Mon, 15 Feb 93 17:43:32 -0500 Received: by enet-gw.pa.dec.com; id AA27000; Mon, 15 Feb 93 14:41:53 -0800 Message-Id: <9302152241.AA27000@enet-gw.pa.dec.com> Received: from rdvax.enet; by decwrl.enet; Mon, 15 Feb 93 14:43:30 PST Date: Mon, 15 Feb 93 14:43:30 PST From: MPS ENGINEERING 223-4656 To: mpi-comm@cs.utk.edu Cc: benson@rdvax.enet.dec.com Apparently-To: mpi-comm@cs.utk.edu Subject: Some notes on MPI 15-FEB-1993 Some general comments from the Massively Parallel Systems Group of Digital Equipment Corporation on MPI: The MPI effort must strive to Keep It Simple !!! The multiple levels idea is good as long as it is clearly stated what the model is in terms of mixing the use of levels within an application. Language independence must be worried about.... C and FORTRAN are a MUST today, C++, ADA, and others will be of interest in the future. MPI must be a reliable message passing system. i.e. The user must not be burdened with running a protocol on top of MPI to support retransmission, etc..... (An optional non-reliable mode should not be ruled out.) MPI should be thread safe. All functions should return error codes not just a fail/succeed status. All subroutines should have a status argument into which an error code can be returned. Groups and communication contexts discussions and specifications must include thread usage rules and guidelines. When in doubt give the user the choice about buffering and other less than obvious characteristics.... But, consistently warn about portability issues WHENEVER a choice is offered. FLAG ALL CHOICES THAT CAN CAUSE PORTABILITY PROBLEMS !!! Be particularly careful with the homogeneous vs heterogeneous environment issues. i.e. Packing verses raw data, XDR etc.... Models / guidelines should be included to address the issue of sending structures and endian sensitive data between heterogeneous environments. When the syntax effort gets serious: All names should be readable, descriptive, and consistent. No numeric constants should exist in the MPI standard. All constants should be defined symbolically only !!! (POSIX inherited a lot of stuff from SVID and should not be use as a strict model) Data hiding via opaque types should be used unless there is a truly justifiable reason. A message type registration facility should be considered as an alternative to communication contexts. From owner-mpi-comm@CS.UTK.EDU Mon Feb 15 18:56:14 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA20138; Mon, 15 Feb 93 18:56:14 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA14806; Mon, 15 Feb 93 18:54:48 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Mon, 15 Feb 1993 18:54:46 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from Aurora.CS.MsState.Edu by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA14798; Mon, 15 Feb 93 18:54:45 -0500 Received: by Aurora.CS.MsState.Edu (4.1/6.0s-FWP); id AA01171; Mon, 15 Feb 93 17:54:04 CST Date: Mon, 15 Feb 93 17:54:04 CST From: Tony Skjellum Message-Id: <9302152354.AA01171@Aurora.CS.MsState.Edu> To: benson@rdvax.enet.dec.com Subject: Re: Some notes on MPI Cc: mpi-comm@cs.utk.edu Please elaborate on these points so we can discuss them at the meeting, in the context subcommitee. >Data hiding via opaque types should be used unless there is a truly >justifiable reason. -- what does this mean???? >A message type registration facility should be considered as an alternative >to communication contexts. I see "current typing practice without any registration," "current typing practice plus context registration," and "typing registration with no degrees of freedom left for user," as the three alternatives. I think the latter is too restrictive, because user cannot encode semantically meaningful bits into types at all. In the first (current) option, this can be done, but there are big risks of conflicts in real programs. Hence, I continue to favor the middle option. - Tony From owner-mpi-comm@CS.UTK.EDU Tue Feb 16 08:18:41 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA16998; Tue, 16 Feb 93 08:18:41 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA19183; Tue, 16 Feb 93 08:16:05 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Tue, 16 Feb 1993 08:16:03 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from inet-gw-1.pa.dec.com by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA19171; Tue, 16 Feb 93 08:15:51 -0500 Received: by inet-gw-1.pa.dec.com; id AA16702; Tue, 16 Feb 93 05:15:46 -0800 Received: by mpsg.mps.mlo.dec.com; id AA14150; Tue, 16 Feb 93 08:15:38 -0500 Date: Tue, 16 Feb 93 08:15:38 -0500 From: linden@mps.mlo.dec.com (David C.P. Linden) Message-Id: <9302161315.AA14150@mpsg.mps.mlo.dec.com> To: tony@Aurora.CS.MsState.Edu Cc: benson@rdvax.enet.dec.com, mpi-comm@cs.utk.edu In-Reply-To: Tony Skjellum's message of Mon, 15 Feb 93 18:57:10 -0500 <9302152357.AA13311@mpsg.mps.mlo.dec.com> Subject: Some notes on MPI [I work with Ed in Digital's Massively Parallel Systems Group.] >Data hiding via opaque types should be used unless there is a truly >justifiable reason. This means that, unless there is good reason, the user gets a cookie and must use cookie-manipulation routines to perform actions. A major example is process ids. We think PVM took the better approach by having PIDs be opaque integers. This allows processes to join and leave, which MPI seems to rule out, but which we think should be reconsidered. Having the notion of 0..n-1 or 1..n may have some level of convenience, but we think it is too restrictive. (Additionally, there is the paradigm problem. 0..n-1 is the C way of numbering, but 1..n is the FORTRAN.) >A message type registration facility should be considered as an alternative >to communication contexts. A big problem we see with communication contexts is that they aren't thread-safe. Their purported purpose is to ensure different pieces of code (e.g., libraries, user codes) don't conflict with each other in the space of types. That's a real problem worthy of solution, but the lack of thread-safety in communication contexts casts doubt on CCs as a solution. A type registration system (compare (very loosely) the X windows color space registration system) would allow users and libraries to allocate and deallocate a range of types. With appropriate allocate routines, you could specify alignment constraints in order to bit-encode semantics. From owner-mpi-comm@CS.UTK.EDU Tue Feb 16 10:29:05 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA25120; Tue, 16 Feb 93 10:29:05 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA25032; Tue, 16 Feb 93 10:27:35 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Tue, 16 Feb 1993 10:27:33 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from chert.CS.ORST.EDU by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA25024; Tue, 16 Feb 93 10:27:25 -0500 Received: from sycamore.CS.ORST.EDU by research.CS.ORST.EDU (4.1/1.30) id AA02715; Tue, 16 Feb 93 07:27:19 PST From: pancake@chert.CS.ORST.EDU (Cherri Pancake) Received: by sycamore.CS.ORST.EDU (4.1/CS-Client) id AA11099; Tue, 16 Feb 93 07:27:18 PST Date: Tue, 16 Feb 93 07:27:18 PST Message-Id: <9302161527.AA11099@sycamore.CS.ORST.EDU> To: mpi-comm@cs.utk.edu Subject: Continued comments on multilanguage support In response to Scott Berryman's comments: >At any rate, I'm very much for having the interface consistent >across languages. This does however, limit what we can do in C++ >and F90. We cannot, for example, use methods in C++ or optional >arguments in F90. Will such limitations cause much gnashing of teeth >in the user community? Will we hate ourselves in a year or two for >not taking advantage of the truly rightous F90 features? I'm not convinced that optional arguments are all that good for the general technical computing community, anyway, unless there is VERY GOOD interprocedural argument checking going on (which lets out most current systems). Consistency is much more important at this point in the evolution cycle. Cherri Pancake P.S. I tried to post this to mpi-lang, but got rejected. From owner-mpi-comm@CS.UTK.EDU Tue Feb 16 15:03:44 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA08057; Tue, 16 Feb 93 15:03:44 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA10592; Tue, 16 Feb 93 15:00:35 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Tue, 16 Feb 1993 15:00:32 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from sol.cs.wmich.edu by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA10578; Tue, 16 Feb 93 15:00:29 -0500 Received: from s307.cs.wmich.edu by cs.wmich.edu (4.1/SMI-4.1) id AA07659; Tue, 16 Feb 93 14:53:15 EST Date: Tue, 16 Feb 93 14:53:15 EST From: john@cs.wmich.edu (John Kapenga) Message-Id: <9302161953.AA07659@cs.wmich.edu> To: mpi-comm@cs.utk.edu Hi; A late pre-meeting post. I hope we can talk about the possibility of adding an io specification to MPI in Dallas. A couple of us (Leslie Hart, ... and myself) have been talking about possible MPI io specifications. Here is a brief outline of IO specifications for one option, a loosely synchronized version of multiple IO based on the ANSI-C librtary. Placing "mp_" before all names is fine (It is my preference to do that in all libraries). Leslie and others have a seek and read/write primative whuich has merit if processes were going to read from differeent "random" positions in a file. Assuming a message is sent for every call this could be high overhead, but it keeps life simple. This also allows IO request to be asynchronous (assuming the seeks are absolute), although we may wear out the disk head actuator ;-). Perhaps we should set basic goals for an IO system: 1) interaction with a terminal 2) writing and reading buffers to files, which should correspond closely to sending and recieving messages. 3) three modes of operation - single_io(), ordered_io() and unordered_io() 4) some specification of conditions for deadlock free use. Some questions: Should all IO be loosely synchronized? Should all IO "require" a message be sent with that request (implying no buffering / queueing)? If not "require" then, should the user have any control over the buffering / queueing? I'm not sure about how to address hetrogenious access to a file. I have seen a couple options. I would tend to not want to force a "binary" file compatable with all architectures. Text files should not pose as large a problem. It does appear binary conversions by the message system in pt-2-pt has supporters, if that works out then soemthing could here too. This is still a thought exercise, not a proposal. john ---------------------------------------------------------------- The Three MPI IO Modes The three levels of routines below and the three IO-modes are orthogonal. All IO routines must be executed by all members of a group in a loosely synchronized manner. All processes must invoke the same primative on the same stream. The following IO routines operate in three modes. Initializing IO and switching between modes using: single_io(group) ordered_io(group) unordered_io(group) Terminating IO from a group is done by: end_io(group) Each of these calls must be executed by all members of a group. In single_io() mode the IO routines should function from a process as described in the relevant standard. On output or in setting the file position the results from only one of the processes will be done. On input all processes receive the same values. In ordered_io() and unordered_io() the results of setting file positions are done for only one process. Input and Output from a group of size p results in p separate items being read or written. The ordered_io() mode requires the p IO operations to occur in process order. (If we have a group order this should be group order) The order of the p IO operations in unordered_io() mode is unspecified. **************************************************************** The POSIX IEEE Std 1003.1-1988 C functions From Chapter 8, 4.9 (Which is a pointer to the ANSI-C standard X3.159-1989 which is essentially the same as ISO 9899-1990) Those functions not mentioned in POSIX which were later included in the ANSI-C are still be be considered required by POSIX. Below: the functions marked with a "*" are non-ANSI-C the functions marked with a 1 provide a complete set, without access to buffer cantrol. Level-1 IO -------------------------------------------------------- 1 fopen(), FILE * fopen(const char *name, const char *mode) freopen(), FILE * fopen(const char *name, const char *mode, FILE *stream) 1 fclose(), int fclose(FILE *stream) 1 clearerr(), int clearerr(FILE *stream) 1 feof(), int feof(FILE *stream) 1 ferror(), int ferror(FILE *stream) setvbuf(), (not mentioned in POSIX, but it is in ANSI-C) int setvbuf(FILE *stream, char *buf, int smode, size_t size) fflush(), int fflush(FILE *stream) rewind(), int rewind(FILE *stream) fsetpos(), (Not in POSIX, but in ANSI-C) int fsetpos(FILE *stream, const f_pos *pos) 1 fseek(), int fseek(FILE *stream, long offset, int ptrnane) ftell(), long ftell(FILE *stream) fgetpos() (Not in POSIX, but in ANSI-C), int fgetpos(FILE *stream, fpos_t *pos) 1 fread(), int fread(char *ptr, int size, int nitems, FILE *stream); 1* sfread(), (a possible non-ANSI-C stride version) 1* ifread(), (a possible non-ANSI_C iovec version) fwrite(), int fwrite(char *ptr, int size, int nitems, FILE *stream); 1* sfwrite(), (a possible non-ANSI-C stride version) 1* ifwrite(), (a possible non-ANSI_C iovec version) Level-2 IO -------------------------------------------------------- setbuf(), void setbuf(FILE *stream, char *buff) perror(), int perror(const char *err) 1 printf(), fprintf(), sprintf(), 1 scanf(), fscanf(), sscanf(). Other ANSI-C IO -------------------------------------------------------- putc(), putchar(), puts(), fputc(), fputs(), getc(), getchar(), gets(), fgetc(), fgets(), vprintf(), vfprintf(), vsprintf(), (Not in POSIX, but in ANSI-C) remove(), rename+(), tmpfile(), tmpname(), ungetc(), ****************************************************************** Remarks: Deadlock ------------------------------------------------ Something must be said about deadlock, this needs to be addressed as part of the collective communications and pt 2 tp too. Some options are No deadlock will occur if: no messages are "active" while any IO mode is enabled (too strong but safe) no messages are "active" in the IO group. no messages are "active" in the IO group, during the execution of each IO opeation. all processes in the IO group have free receive/transmit buffers during the execution of each IO operation. Some type of inquiry function might help. All processes in the group may execute a probe to find out if it is safe to call the IO operations (humm....) Remarks: Sizes of objects ---------------------------------------- Restriction on the sizes of objects generated by the different processes could be made. Possible examples: In fread()/fwrite() size and nitems must be identical in all processes. In printf()/scanf() the same format string appears in all processes. These would make implementation faster and easier (and should be true in single_io() mode IO.) It does make slightly irregular data distribution harder. (Why can't P always divide N :-) ). Remarks: Buffering ----------------------------------------------- There are two types of buffering to consider, queueing multiple output requests before sending them and buffering the response to input requests (as the normal ANSI-C buffered IO rouitines do.) Buffering works in single_io() mode, but it may not make sense in ordered_io() or unordered_io() modes. Queueing input requests could make sense, for non-blocking requests. In our distributed graphics system we have some asynchronious commands, which are not loosely synchronized. For these commnads each process gives an independent stream to the server, which it must de-mux into a single valid stream. Any calls which change "state" in the graphics server are loosely synchronized. So the server dose not have to keep state for each process, just insure a stream can be interupted from one process at a valid place before switching to another processes stream. I don't think that type of IO is needed most. Since the most heavy use of IO would be the ordered_io() / unordered_io(), and that would seem to be large size chuncks of data, buffering multiple IO requests may not be needed. For terminal IO, the usual line buffering should be OK. What I'm really trying to figure out is, if setting up things with no buffering/queueing specified has a negitive enough impact on any applications to worry about. That boils down to how applications would use IO routines. Remarks: Higher level IO ----------------------------------------- Mapping r-dimensional data onto a logical s-dimensional process array is one fundimental high level IO operation. The ordered_io() and unordered_io() calls above allow a fixed dimension data file to be operated on by a fixed dimension array. I'm not sure how often dimensions above 3 are needed. That is, how often problems of higher dimension can't be well handled by projecting into 1-3 dimensions first (for the data/process distribution). If the size of the array was changed, a single_io() read of the file could be done, disgarding items not needed. For non-parallel disks (from a host or other single source) on an MIMD machine, I don't think this posses a serious problem. For high preformance parallel disks, or disks distributed on a network, that could be unacceptable. The only way to support that type of IO would be to define the process array and the data file and let the system sort it out - as well as choose the files internal distribution. (As in Hart's system propoasal). The EXPRESS model is successful and should be considered. Remarks: Multiple process IO ------------------------------------- What if anything to say about multiple processes operating on files? limit the total number of open files? (eg System Constant 12) limit any file to be open by only one group for writing, for defined behavior? allow any number of groups to open the same file for reading? (requires the server or group to keep state for each group). Remarks: The three buffering modes in ANSI-C. --------------------- The three buffering modes are: unbuffered block buffered line buffered by default: stdout is line buffered stderr is unbuffered all others are block buffered The setbuf(FILE *stream, char *buf) call can set buffering on a newly opened stream to unbuffered (buf == NULL) or block buffered, using a user allocated buffer. he buffer buf must be of a fixed system supplied size. Useful buffering calls, not mentioned in POSIX: setbuffer(FILE *stream, char *buf, int size), allows user specified buffer sizes to be used. setlinebuf(FILE* stream) puts a stream into line buffered mode. setvbuf(FILE *stream, char *buf, int type, int size) sets any of the three modes (using type) and allows a user sized buffer to be used. setvbuf is in ANSI-C and I suggest it be included. The tie in of the buffer and sending a message could be made, implying a message is sent when the buffer is filled. Giving the user some control. I'm not sure I would like to state something like this as a requirement. Remarks: non-C Language IO ----------------------------------------- The functions above are clearly C oriented. Compatioble calls could be available in other language bindings. No attempt to specify native Fortran IO in MPI is suggested. Remarks: The varargs forms ----------------------------------------- The varags form of the buffered output family (vprintf(), vfprintf() and vsprintf() ) is in ANSI-C (it was not mentioned in POSIX). They could be useful in avoiding extra memory copies. There are not corresponding varagr buffered input versions defined in ANSI-C. I would suggest that they would not be as useful as a stride and iovec version of the fread() and fwrite() calls. I think iovev/stride versions of fread() and fwrite() should might be the first extension to the ANSI-C standard to consider including in Level-1 IO. The format for these would be compatible with whatever the pt-2-pt calls become. Remarks: Blocking, Non-Blocking and Synchronized modes ---------- Some of these calls could be used in all 3 forms. The setvbuf() call already could give a lot of control over buffering. It might be wise to only consider "extra" modes for the fread() and fwrite(), which would correspond the modes finally in pt2pt. (though if extra modes are implemented here they should be easy for the implementer to add to calls such as fprintf() and fscanf() ) Remarks: The modes in the fopen call ---------------------------- Limiting the fopen modes to read only and write only would simplify some things. Remarks: The unbuffered IO calls. ------------------------------- Although the POSIX IEEE Std 1003.1-1988 C functions do include the usual UNIX unbuffered IO functions, I think we could take the same approach ANSI-C does and only specify the buffered IO functions. Because of the fread() and fwrite() calls no functionality is lost. If the unbuffered calls were included I would suggest the calls readv() and writev() be included as well. int readv(fd, iov, iovcnt) int fd; struct iovec *iov; /* pointer to an array of iovec structures */ int iovcnt; /* number of iovec structures */ where, struct iovec { caddr_t iov_base; int iov_len; }; From owner-mpi-comm@CS.UTK.EDU Thu Feb 25 12:19:04 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA25996; Thu, 25 Feb 93 12:19:04 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA06338; Thu, 25 Feb 93 12:17:37 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Thu, 25 Feb 1993 12:17:33 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from daedalus.epcc.ed.ac.uk by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA06280; Thu, 25 Feb 93 12:16:46 -0500 From: lyndon@epcc.ed.ac.uk Date: Thu, 25 Feb 93 17:16:32 GMT Message-Id: <2150.9302251716@subnode.epcc.ed.ac.uk> To: mpi-comm@cs.utk.edu Subject: background information Dear MPI Colleagues I found the February meeting both enjoyable and stimulating. It seems to me that the question of process groups and communication contexts remains fairly open, and the committee has taken steps to move forward. I am sending the interface specification for the system which we have implemented in Edinburgh, called CHIMP. It doesn't contain heterogeneous support and the such, but it has a very flexible approach to process groups/contexts. We've used this extensively for both SPMD and MIMD application and library development. It's coming in three parts, first below, and it's a uuencoded compressed PostScript file. >-------------------------------Cut-Here-------------------------------< begin 644 interface.ps.Z M'YV0)4) F=(B")DW8LJTD.$"AH(2)8;(*1.&SALY.D"0L9,&SAP0-5S0R%$# MQ) WBP*9/QX!@7&],XE4B13IHW;HA4I KB2ADR((R4$0,3!@@8.73(B$LC MALT<,YQ""7.FS)R,-/)"W-OWB1PR93"""#)G3!DW9)P*>5,'C6 M;8T<-D#@H"'#*9$W8^JT>4S'"%@Z?T'L95/Q\IL64MZT">-&-M#:;FY/9H-V M-O#;2>@ ]>FT".23;5:[@?V0B)4D Z'OALSD,MFL'C6BQLJQ^O6!1^O(<9P1 M!)4R6$!0I@.GSLN8>!LRC %#!UP9.$A6QAF702$':E.404=&=)2!APMP'/A0 M" J\\!X61/CTD@PUN-73&"\A9H9[\&4((@@)$=C;"T[L(6(?(H+P@A![B'$9 M6B\Z(>,4>S@X!AI]Z/@"%GM, 8(3?2@@A(Q42+$''7+P-@=M#08I8QI3?F0& M4',4):0=@:51&HZZ,2 (C0%55)QSNB7I M'(@J"I8"8(I95*&-@I!F4$T*.N*HI5;:6Z23FMI;JF."T *KF,($ D]HO>J> M% KL!F4:>("0FAP33:?L2LV240<<&ET+ @T@]/72@96AQ:V/:(!07XC:6HMM M2=Z" "YD((% K@+G7?2RY[W9U &5& M&6?M,8=R=-0QQXJ*.6*=XF8RN34=%'@"#,,.18+\F\[)X*&#&='(TN_-K/;_1+,]+ZCCQ M2IC)*$8878(0'U-1YB&C$%')'%Y\,]91-1E#H,'4'@I$SI00CA.7F+[.^=E3 MWTFC_1B.UPILQAMOV$O'Z=/UD1/A81A>LNQVR T"&V_P9#/1]-HG8]-+;_%R MU +/,:)=0KX-Y=UKHQWUU&%4/;P9<[;Z@O&"RZA\W%M@[Y!;V&MZ!M!"/T\U M&R\7L0<<*(& M;H:E6&&=ZR_U.7JK[W\8PMWI$$&'6C8P_[(4)$P* !^4\$, M .,UASJPQ5OZ0T,+T% &EKA$@!(DH'*RA:T$GF&!W&K@ Q44P1;@ 2EFV$,, M -2G#!:0@[=[S ?+=381=DM!G')@"?. 0@RV0(-A@*$'%Y@3&[9KA3=0@ A+ M2 8\^!"(0I3A NUB1!(J;']IV$U?GOA"^-&!;&6HT-XN\T$PNB]@P&O7"X ' M/+L0ZTBR>IG_GO8".=R!CNNAXQFP9;0*C8&/:'O!$>CHO4!:3@Z8>UQB)!8& M.Q3E;'8Y$!N"8B3XO>!O@6M73F[DH!O200%&FE'BPA">=AE)C0-\87S8AZTU M0NEI VR66_9W0C.,:'\\M*6S)$A!"W[2AK1$X2[YY[\%OA&7PKS?3WYTEC)P MQ#'#[-__RK6_7K;D)1^C&= 4@"=?N< NP=0E,"68RQ%]JXA.'=P M0XV6BH:F+O0%4T@"IQKI)1"TB26748">VO &1UI$1E9P546HY2SU1(L.^*J6 MMNS2+O@9BUPVJZNVC$4L!;C@)K>C@\"B8(4^"$P*A1U4&>14%-SI+D:L-!=* M]%6&HF;5GA.[2%'F4-4[P,&=(#CJ8314H0*OD6BH^_(&!AX+U M NLD08O+#1V'0@,[P^G(3B#@4 T4L(?0&8F*!1T?_-QR-DXVJUYCL$-O0G8Z MI]F@!C? 00Q<<(/06(]718G/??.[W_X>D+IH"%.O7"8D.$BLL_J2PMJTJ:=N M!DU(0CC-AFP OO0&4@YJZF0?ZQABPQE-1G80&(BG8CC&K;B3\2FLPJRPAYZ P4G0KG(^AC(Z.H@21]UPSVXY0:>O$R#Y& 'H' 0 M!1F: QQHDX<4E"$.\L+#CVQ& P5X2\J)J3(;4. $^%#!RV"^B&$CNR7&RFJQ M73)LG?.L6,;*"S)[V,,9YL#58WGS3M]40).$1<6?3,5=(6[6BPVWYZ)(N")[ MR/ ; M7.OLRSGF>82&;S:5A!$]J1AFY!HE5MET6_6-(EYI2CR22L2NMD;9G6 M,#S=^>ENB1J?[[RSGQ5&6,H-NM!0DE)]V^ NRL +K^]""UZC$B4W&.\BS":K M65/W:IO)@0W>.:M#)GW&\7T[W-R.=+F]#6Y-G75^D_3U/37[LC#L0=L*>L/P M%KO!$9_/1=IZ 6+8L,'XN$4.^+;(RYI0)'/%C^!!?./YAMIP;$W\BGMH ;>: MD"2%D2'C9^.XP@*V$!"(7'L9M\O)7W &)YO\90%4N1SWD).5JV$/(7_9&O:P M\815Z XN1WA9\_TRJV%+XR"XP\O:L#Y?G4WI"I/JT7,"=1F]H>G=K/H+XM!T MNVA=#D4.;])?-B=L/?UE@L46M[3N1+<82>C;?ED><"YV208%6V$8ZAOFA-6. M01"$@J(HDND(10- MZ&MU3.'\(%._*"JPI#?5VDMXK$(0N!=L22#"&<7NW@I5P0UIB,/!8C][WGS\ M1>MK'RN%G>?Z H4-VGR]6T+33;N'N0S^= M>Q]M\1 7EN+52;G-U=2E'&4M\ MY'M!$Q2TDC%\!*\QVO'GW:" SOV8C#T2<[F\:,:!36= B1%H41!JY/:9E"WK M%4/!!P>&)3Q]EF>4$S87M5;,DEIR(C,P4 ,P:,R 9$(#]AH":P(3 )!@(VL$*!!"8*< ,X$4A5 M14=VD#HG]@()YE.OV!C"(B2\,3ZY. 1,@!UXU#\)\A@@8&M'4T=HD ;)V!O, M*"-300=35F-"$AP-@HTR$@8TP(TO,!5G !38^#) T"E341E1T18C!@2[Z"E[ M<(ZSIHX5Z"BZZ(LNHS! ":S%H_H6 ;TV'THAH_FF&"JHHHBR#C$:(S(8XX M)C!@LEP+"06_:(YC &X.]@(3*7:!-S)594OWHXJIDX\5LH\H%)(O<(N"8X[! M.!4"$XPKJ8]VE 8J-%WV.)/+-9/16# @AC R^8SDQ1\"Z8QID)//N),? T?Z M.$E.%(YLT"SY.#*3-'=.V6+FJ!Y-B94Q69+J095=&9,RHD7_R),'@X)_AA;[ M(C.'2 =5X&"UU7D:@Q:[U5MO"2(5T@9HT2[Z!S4:LI=?9E1=,EHG0HEX&4.G MUQ:I5YB[@0=#5"Y;%)?)0CI*=I@RX!9OE%2S@X(+=8*%28GAJ"#7J"_:&##Z M(H[DJ #Z\F-/@@?L$S@UTP9T,(1;@HIQI1')9FW+9A3#HAQN\ *?E3V:XED5 M04T8^5D E"Q,L7.W>6T(!R?+F9L)IV^&]9S8YIS5QIS,UFY$9UC,EQ-V%U#7 MF9O=>8-!09T(QXJ^XIW6QY/*5IW4UI[G&1KJ.4E>:)[,)I_DZ7KVB9[S&13/ M$G>E=I%[5P;&"21]4* :R 8@50;"N4!^4AG6J!O).2S0$CO#,FI%P7+^A =1 M(UG8A+J<3).HSRNYP:.!%QF M0!LMMQ=2)@2T,0:O-2CZ$INM&3!P@ LE!X@*78D@=F.BR %EE3L*#CXZ52 !,X<'";-A;" MTDU.&B$UV"Y/ZC1TA4/@9T/@UZ=BQZ=ZZA#M$J@G!7ZQ$E-EBI$@@*:0JHUJ M^G%LNFYPZJ%R2J<4 S)ZFA>&ZJ>>M&Z"NBENT:@C(JD= 0*/NJJ4ZGOKHZ=_ M^A*#>JB>Y*;'$D=[<*48.2=?*GSM(P,WP*EV&J:R-J:.FJ:JBG> ]JMP0"_M MDP/$VB!W*J:/1JAG$ZK"DJB;4JKC0Z@YH:UNX2VXZJVYFJR0NJSNLVF5N@?. MJJG2VFR=:BZ?.JJ$.JOENJBF>JZIFJ:MBBVOVJQQVG_8(JS3ZJE.$ZXX1*B( M"JC=JJ_"J)NH2GR;-6MBD >@$:M.(Z6Z$UGJD2=]5E]82[,]FOTIB.C$T=$FHW8QB7]N'VV M\B@C&RS8YUSL!RR9TJVSIE:#UH_6A5U0.&L+!3X*$FJQHR\>D11S8B0#IE_\ M90,'YBA((B2R!5SO@0=&P!("$[8D, 5<1093,*^GUE5U^1*+V9B9R&-WR1=C M:6LQ$AP(9[2>TE;0PAKL(V6&YIO&0P)?&;:P(2-TJY66*X*2:P8D,)7F8CR7 MV[F?^Y2A*[:8.[F&L[FIZ[GO);H?X8*.^P:02[ICT&(50K=C )6M2P)S\+JN M>[J4&S?"RY0Y!(#!TF"3.P>KJ[K"VY7T,KF@:T.MHB-T^[O"R[S'>JV3N[O9 M6RW!0K?&:T.ZFP?1Z[G:^U*3^Y4VM&C7D[/N!(Z) 3@$&K\TBQ)D8[\-5H,B MH@! H):3UY;Z8A48N3[2:[KBRP;-Z[F:N[YR(#LD.!V/.QW\29X+VJ#EPBV9 MR2W]Z2I#=U;,9[+Q9B3:>58DC&ZZ6<*I(Z!=4J!],JD,6J >''=",CIO&[.A+?SZK/YQ"8\IHBB>)J;.(GKMR.A.(H\O%F%IB/'AFJ(NQMR,K6R M*\&T2\%-XK^@N(A 48JS27"QZVP9@QIDO%QD\* 4K,9T@#&R):1P%%L*0L2C MN <3J0!V\9?!*<-C]<&ILXH'I\(V$Y'GYFY__!'@%\@HS,+U>YP7V1$+58LJ MN6B[" +\N+U% 9,'\L/("#%2A9-:.;XGU2IR<)2>3#FE7)1?.;VCC"F&I2X4 MB\FW8[K@![J,*@6&EI'%H//,OF2[T6^+)3:[RUK,#K-K$8:!+%2)$Y MH<0:/QC@JH3N,@P=1@A:,L[>U&540,S,*/03J'0/LO=[M_=[N'=_T#=\.,=_VC=_N MC=[H/:R?/=[G+5W=#0-%\-TW4 0Y< ,Q, ,^0 ,X$%0\8 02/N%S.N$4+ET8 MGN%BE>$_N$@;N$27N$B/J$#+ETMSN$][MT[CN%!_N,\/J<8;N1$CN0;KN1"?N1)/N31/ MON11#N4^+N5!+N7>K> V< -$@!=X[.+8'51" 3;^1 .B+0,XKMV??0-#<. ' M/@/J7> %'N P( 2R.!HQ8 ,-7F9$P ,W,.'29>$D/N(@WNB.#@.* M#ND7?N@2[N/\X><(7B9H^^)V4>C2)>4QH.=)+NH^+@09'@.FCN$S( 14ONJM MSNJJ_MU$[MZS/@2U?MK<3>L"KNL$WNN?[NN2/N02ON.A7NGKS=T^;NO#>NS+ M?NBA;NL2'NKJ[>Q&8.MX/ ,Q<>@R$>@O7A*%3NE(/N$[3NFB3NOEKNPF;NN? M?>SK3NOMCN[,[N-&L*+E/NSR;@3$/@2AGNSAG>[A;>NT'O /_#'+O#L3O#= M3>XD3N\"+N&V[MTRD.U&L.V"G@3;'>JRSMT\$^Z,ON]&,*ZD?MIX$;_ ?%?7Q_>NZ7O6]?O7'GO4/C_5>7_!13_ A MS^P*X.\?K_'&+NY+C^0+'A.@7?0^8/'D#NT3KNQU#_9X+_5#(%8&G^0_O]ZZ M[MZ![_#W_O#%;OCZCOB!__=:W_A[;_5=O_73KO86/OGVWO8)/O%P;_&5G_1* MW_F._^MX'/FA[_>GGOC)/OC57OBL#_B*C_BGK^&\OO7\;OHQ?_:)OOJY3^68 MK^V;O]UT;MI+?_OE7N#S?O+0?O)SVNQ\/O/++O,P_^LOW_+A??:3;^OJ/0-G M+^I)U/S,7_,5COP[+^3[GOND?NC+'MY%L.PK-/04'_='_^R2+@1*?^AXO/JT MSOXY'^_\G_].__] C^DAN@!X[F:>_UMO8D7_&<#]=P ;( /I_FXW:<;]A)N!55!-3>NJ-Y(/ #BD D-P*5GPD,@20P!9[ $DCSQ K*TX P M\/-ENS#W^PA?[OMY-I#VZ<#9QP/_WNB+?;:O]@G!'$ R&-R& QV#=JA.# M9] ,]@PSF .$P,AK@V_0#8(\.#@'C8#-*P)WT.;1/#WXV5S@S=N#?S +7L%! M* BCX.\[X29L+UM@G=6R=D M;_IO_24[4;C>OERR,X7K[0;H/U5XVMS;#-!_KS#9Q4+ U^R,0/F3<#F ^^FY M^$-[HD[[*;T :/C*1((C>ACPZ.$^M:?Q&-ZQ.WONSM^- M/;V7]\2>V#-[)FX %CMBA_3I_N"W3]$BCZN%_J^=%@0 M"^*(8XEZKB#2P\$8]NHA C2,A!$Q%L;$V/>R'Y)SC'PN_241H[@(#2+;JP%N M#R[40&6G!0DA9[2( -#=^;@_N.X4 !\DC>GM#=K!.A@'VQT<]'&OT;O%QIC@ M!F%C;92-MS$F++A6^ 53':13>4ORU&_*S?J,ISV\XZP3L"%QUSW\Q3Q?UH'F_=KOMYKD[5C$-\ED1WW\4!DVQ-6F8[;);CD*.64HA5DD'$101(X$/GIC)U8 M>8@M\N&)NA+IZWYBB.R)2/)(*DDBJ>>,7)#\'@^Q Y*Z&V?X8.+;2X<-TN'M MPY#7XKK;)@1VZA$H"KOOX?P2'H@T?N[Q['U'_H!$PIRF@W_"D"YZ1YF7_IH? MD=-P,^_?I3\)Q_X&()?DACQQYAG)0=G\"*6@]'Y0LOF%0WU7[WJ>BN1^<:Z] MP<2]:/34W^X[?%QR_27#9H"O%(;';BL81 M3@:_AX?^6"2]BV^>,"I&0VF9#:>EX,.!N*_8)3DA,.2B/S(Y(["T?NV&2*"W.KTNCEP)E9%_G>KAR( MP;)7XDQ@R3._A\X4ECTST@G-"R?TSN'[XP\!HM!-O"'P CL@W;-W(2^_9#\\ MYQB'5=4TF)$0:UK-?KBAP"9A,/XKG,MD) 6[V< M.YA^\-7MXXBZ+U,6PVY8**]AM;R M9)%2:!Q)'Y4BWB MQ=PWY(:G=ZL+L*YTRLSCER8K(Y\DEGH.";9++#W*!+?P=/B.:3H\2HS=/UU( H< 5"P")')( M?KP65_2I+[+D]O6<"I(MI;P!:N *:)[UG WV@=1*">K<[61J1 MW0?T;T@NOZ2_RBCODB+;*Q.J<.*U(J.'"+.NB7Y+"=L)IX]=(^SK^O2"&K(R#\HJ>QW^%!$Y<+A5],L(.V MT8P2 3]W"=FH=UNCE="-_L8=]R.!8T:THN23A +10Z?][)[\^XWED5]"T>HW M1:G?/W1R!'*0W@!^V43/XA!5A/"Q N)1+,H7@>@&S(6,$.$=R/0W'/DCL&ND MC31$ <@<,$IYWX!4$D-1YJE2'1G]F"@I)7)"H#SB/D*JZAA?R?)SE-3HB<,J M>/?FGO-;?DL1V3DY?CGENJ.JTW)4;F!>.:C'[,S>W;-P3Q/_1;PKZD//',R( M##6@)*R0-K?=KF+M$WY[3KJ<3!&X/=,?5#R78O)W KNP*5W.J614=:J.[?V\ MGW=.P2FO>WA!( 80@1L0!&Y";FMP#J'043EZ"D]'HQ <X)3%LA056 ( MQ'7OM)W^SL!' C'<.?UY0W%5[M,AX$]I@+)TJ.ONQN$ M!FBI"2^,XV,;APJ M/7&8YS1@216IZ@ZDACO^P!4CGD>5>P"SVB%.6Z?O6&>O>W@<)@C$(IE0%SPJ M#3B?%^_O\'54ZM4=J5]. M:+43ATRSK][5O;I(EY\7+:S<;;#R3X='X=AD"'VK-Q7XV<]@YPH+G&#MJS[1 M)][/4,E.T:EGO9^9]=.MNYYZ+5^@^/N-JA#!M:+("B\?8G76W02DC99RN8O,$GD#F*EUOJW=]@2F5:!97$?I/,^"GS*N)U;"B5Q2( M0?U;I1.'8M-K-M05*%-SGWV=H#$@M;K5X]HJ0QZ'&7:5];O95V%CD8W%LDP6R3O;']KJM.:STW/4KD,\QB:C* M(_LH'V5=#:S_+6#*U_06WG) ]GNE ;*/5K]3.37;;'S-FJ^5=,=EV.*W+9^]@932,6#(-J$ WV M1DCK!7M&I66#UUMQG+^5YL MASV3&M+4*4-9ZT2A(:T=@V2VS.;:$+5K2:DR]+4G#]CVN_7>VJ7;70$EKT"F']JOJ\FWUV M!')70:M2B5VBS;,T5M@AO?_V\+XLBXV*7O2W-MS@6OUP'WQ5?B-0USG;I4VM75S$A(3RU["_B/=;CNMD@'*0; HSRK[9+#A/RWJ=\([8X M5ADJO1]9[^2I=OQTF[7<-DFT*6<1W?]$K8HVPRH\SY<,7>&LG+!--[ 2.%?H M75UX$'LAS)VT:+:07@:\V9N+;5GU]2JW;1K\\SN'\R/ M2-3G&5/BF%]!&Y=EE!4.[WI8.FHMQ^JU3'VO#_!Z52K7544KX76*6'6J\KCT MEP#KI%/UCD]UJ3I.(TR](ZCCNC]39]K,8%D0DZ^EJ[:^5QV: MU&^KW]Y;J 6!G'2B_LJ.*V73Z=8;K.O6P?;+&(@%X^V\M7?V;LAUU@1K@#TK M!;; ZO0"JT<%F8'5*00>AU]PG]Z$5:EGL6!#U, =.$&BX J\@C$P"^[ 'UB] MB940K.U(L _ "Q#NCZ*].SG@3"*XO)G#4R-R26:9!)G@U]-UPXI=5MS5&6-I M:A'(<_]T1BK-I A_=V63J\+9D@B,PWW'_K(?OWMWBC$,2[V2&0^?G!FULI>P M+A)='Z=R[6Z&S9*9-_&M.^QZ3CEKJ$RHSE#PF>!3"4"YVQ.VA9$U^"I?5HH@ MTZ.!A',*-]_Q2--FY&@GD7S$KQ,2PTYD9XC)&W#ED=).]Y5-@*=R+6^KO)&5 MCNHRXDU(;,WM_A5WG14/4T,]+.GX\&_TPU X$$_/08C?/AL2G*^4;O_FS4$; M<'OQG\VVA,\6KMPW7#;E';V[>6GV6_(Y)+SUV.W)4\;]3@121B4L0>=DMO24 M#7<8!M,4BSF?X?>3G.30_GHW9!F+X9^X-(A5.&=6X0&G3[?PQX.*/VY= L ? M*(_[WP,$C?9XYOU%$YSGVANV6[5'EKP=Q0Q:43TF75R.OI@7_]E!RT+!*]HD M?)9552HX67Q.Z:9%-.+2;FIA?@C>GEP]AKOY^"O? MXWI,CW??5^S'5-DJ V(XR2ZU9=3,FO"3VYKB=#N7'>R\S)UO65O.8EL(B]&1X,2IFFO?CSC!(U,*(+UNF8YYYA0WB+#V([0\' MJ&7O:_P2W?K#??CS ^[/H&@066GWS)' DQ 3O]5\?S5>.3:?,6Z[_636.>)H M[D@=MOWN_*E)8V(5U"CFR9V_?ECI%F5 M_;*&3:O?EZS*9X^(??EJ]36\%A6<2M_%RY_;KW]FJDFN3+3#JTQO>_/S_(/< M>,6666(K2HD=*35\M'?VV@##%P0F=).KT+B7&E/? OM!::K95+HG-/A^Y^*[ MH?I6. M/$X_S\*35R>5'-\#=>]7'7=7 ><4U2_S/75EXG 2Z%T:BR9<:(W+%U0 G\ MS.Q(7BT\M&S2-?OE]@,VBH)^V0*F#23( !7TX-8T@# ;=@+*R("<$!-@ IIV M"V\Z31]5F""G:0:4,6T*X$W' !54$M+TG+;3'*(MH&D1-:AGP-E(TX.Z+L#I M\+(%.H39H!DB2E)+:A82J>V"4[73(>I1$.KP4A(2M:DNU:B:5*OJ1#VH75%^ MW18U8 O4F*.J #I$0^!P,L(X>(7@T *& UJ0 V/("!P/D% 2PG81DQ'A2G-Y;#*,@S&1@(G'[)<^)T2 +G<<>M!;$?%G!0L;]8U7:Z(84? M:9U33!- X[$WRA6_8 WP 37@?%I3(S ^KALW_:\[#RHB01H YWSJ$(#7#Z[0 M<;GJ>>H&=@(F 9B: M4K.Y\"*B/G7,A@DMFS2 :OE4J1_<#%!!9\/!@6HZO064-FG 8[%Z+O#LJ3T2 M9+7K.!N:S04XZJT=&EZ KK8-O1I1_.I@793.]*&.UWV;?;9K! M+6KVMF#3H-I.Y #&C9B.JM6> >.E+K@ &\"Y.?>#\]J_859N#01(. MAJ>U:9O."W'[:+?M5FV[(?>;%MIIVG&;ZKQMM&=VH7[4C1MF%^K[9KL;=Z'. MV];;;.1M[2V^61'-P--K.WV'%_6MM].WVE[?JEHH+6\\3;F7=TX(459[D6Z+ M_=*Y/7>N#MU@.S=LAXP6K.N)V6[=;MIPY^[:+;<#':(NW*KZ%<*$YKV\RXSN MQMUKVW@C:K]]M!,UI#[<39MP4VH'CKG-QNQNU-M[:9MOH82S]_;KAM]Z^WQO MBY9=8T*UYN[?-N!S?VU>[:L).-@("@MN<@LK$VXV##<-* E.6P5Q,,B-!$%U M[P[ABIK#@(1*[1#6M"RZU&_:!F3J2LVH@S;3%MIPVRZH.:M=IUFGUM[:GQIT M4XD +J'8S^F&;))-DKR!,T"(W,#DN19.02U4!/5 %E0 TE(NPH\,F(Q(9-G6 MCUX@)8EA"JB*C! $RHSBP5L* H\/!G>2$?PV'K,91 5;W !M6L(-2!P: HH< M,5"P/4 "D@ 9R @$JI^X@(F1!UB H]@O+/LNS( 7 -D "V_VS5 !WPV'2"T MQ8 :< -WI QA"Q+0!U"0 O 57PYHBZ@@ @2&F$4#X@O.JQBON7 V^(4"J]5- M' AHI0'C5;Y2#G<+;:)_>)7[\='J^&E(#=*!\J3R,2 &[@#N. -OP 40*!$4 M$ 9"03@("0%/- 33@!I4 VM00+$!"2P6L^(3# A$@ I2@2P4 2@ @&$SZWIOD?2$V?!2(,!G"!1G1 M#)4\FE-SD-#$!\PCUPZL@9X_!*' T(? 7; +[SQUH Q8 +N0 J0%W"@G[3S MK"[//<)#4 58G[LD;V3 !) 32)M,*"8]Z\7L'=F35OG.=NB#R2 _E4A)@)B MN.MN0A [&&+CK.2&N[W7Y42$*.X&ZZYYM6]R P<[8Z\A9..QN/9DK M=L*>*-8"4W#K_/HM4'8ST-B9PAUHZS9@L9/V2\(4T$!;U\RJ_2'XIO,!V6^ M"TC4FYVQEX 7<#Y.>P*P 2*A!L3VV1X]V$!MU]HE(;>O]MZ.VI'[<#\?KCT! M8.[8+D:H6!7S% D@OU#WZV'=R8L"2 O0*"S 8+^$Q+ Z(D:/"@!S 'K_H_4 M^ZRI'ZCI@##V"D$;Y$!?\.[@7;R3]S!@WF\$?$\ ]3VHV_5_!-^3!%^_))V] M&DT9_,Y9>$M\/_!^7<$G!OR>E'X$4W!A!GZ^"YPH<0<<^]^0 _A=%]&!!$ N M8CMXOP-D@,3+'^J> %!\ H !":"0Y9OO#N-=0HQ78=\=Q9>;%W_CT M%*(3:BL#7SW"@Z8)OS=22QC1\L,IHUUVJ6814-L8"/-G0<*#^#*O@]#Z43M. MCMV?>'D4\.83@&.7\TH$2IAY.U_/T( "<.S88L_#@:_^Y\<\B(_IQ9W-/Z+> M7C"*^_SA\!Z>*1AY^H3F\7QG__"2_M";=A31VC4"IO_TFOZJH?508WC.? FX M\R5%J>/@Z+0MAKA3Y19DX-0?#%!/Y\]\@Q=.]_T/089S[L^GPS''9_[BF0-M MDFT37H=7<>9>#)J#<^,5"Z4Y,G-P#D&;$Z\;D,V_TB*%,N&\*)VC.?;,S7D_ M3^>HO Z,@5\#(>BY0" (!@$A*(09T!"H^A0P BV WCL$B, 5-+IB, (3H0P@ M =\# E %4 (AJ,)7(25<'Y> E" \W["VA@,$'$64(3A4 )O WT!LR1T$'\ M08\*4R$CK/M[ A.^29GA]QG=5JQTLG #7@ .> &K$B;$ !V 7X9%&Y ,SN9S M9 :[D=39R[#:%G/J431Q%+ 4/D)E"B^L$R0DP M&-))5%@45@8QG(^;P$-6=/]Z7_M%?+,B!>D)4P@538!GR01JI"3;0 OP$[E /6X7@N+8R M0/Q;O\Y'$7;#NO(2V'^^7\5W&9A_EHSN-_,9!0K"&- &Y %K 'H-!P5: &E M M@;"$2!9A@JU 1B" N=\W,0-D!%8 XV^Q_ M6@(-(@),?6>? ,<;B !%H.=A+H@:K 'A5R%,,5:&].&$98=*8@60'\C"S&(]PD944%18"D$$9\@'/,W""FA1LY7(2@%9LS]YS3P M+])@P7 'K#"]'_[7 @B!E0J*$*2L 6G=_$< XG^X7CEH!LQ_!HTX:/<9!:I" M"S 1+'E4RH$0^24M$9^@)R?("R[ &=#_7036PD10%A03Y4+W%^ U%@\=YV(, M*AA%00\(_:D0@?Z""J%]6J!5NA:>?:>0"S 1.%0YV/\R M4P:X01;N@TY %G@&?@U;0!= H5@:1\"2H&8<#!\!"@ *#=IASK MIB%BB,?A?'@@%G$/SA'P(HJ!)>(]&&^@B 7#,_(%%@5_ WW2(M8'D-^CT1YF M-B7?=N@D;@=,HFOH8QXJ]&']N$,P"UD 8L%[G 'O "&03!R)0:)18&6F _Z?C>*IK 2 MNA-BHAM %$P2O(69:!ZZB:PAM[B%PB )A:R MD M0%!WS(6$G&)(^/Q)?P0?9W$R$ =]XF!X%MQWA=_H!\9X!ZJ# P@!2H \8F+0 M!GP$2(&[< :P!3$%I2*#: J>( 2#U[P$: ;D-Q%,&>9?#T('N(H18._G- @9 MB4%1$-!LBM+?'G#6-( /8+*(.X!XG:*T"#\()5\&"&"S7("WH@28)'2*V>+_ ML2T("\Y'7D NKHO>8@WR>KP1X$< 0G!XBY%%7.=ZC B0A 7#+I*+Q=Q TQ5 MB_LB^@#=D+EE0LEHW=0U*@.VHD:D,6 *;A95!/ M*!,%8Z^'FAQX!X$0< 6,C(R-5##E080LH\I(2@2-FH++^#68AFN 87 %/"/4 M"LUX+>(83$T8(894C)^%E$$Y 'EI7A031CB-%\$R4C4H**P>AG(S:G@CW\20 M5-@ N%[GP=>5C6T,$J,B5(Q>XE30)82)E,/S%S=N%AW#1[ .5HW301#($ Z& M=@$*P 0@ 2E 32 #U 0T 4A0$YP-+$"9L1 =U &:'#;A7H+@6U7$I $Y\1" M0 -,CI%0O;<0T'M0QDP@$C NVP8WP>9PCINCB((GV'9/1@Z0'-X,(4%.H I] MA:3CC(()!<:C0.^ICF7"Z+A(_16U8YO'.YX39P+MB">\AE#& M\)@3+(_U'K>WN16/C\A"\!J**)H9?W$[^'B:X^28 VR.;L'C^#HF.)ECS*8] M2HXJBWI0.BZ-U:!A0"4FB2 #)_A)P'IH'9Z XR.THKVZ.79C[1C<_@ZKHZ< MHVWWX. )(0'V:-MI,.JC[9C9$(^B8TEP/(8&9@N4<3RZ!7,*[4C3&(_\P>LH MBVB/OL+]*#L"=W:!KV#;Y8XA 91Q03XBR!SMN,E-CB(D\;A!AI MP/WXX%20 M#\X)R4UDD!0D0_"Q!(]VP6A /!Z//21PESY2&0KD3"@:N(XY 8Z')]![$*3K MZ$ N.--C_CA YC;XXWZ!0!*1&.3F^."L2MKC[3@^;HX2)!6Y1<:.U.,-"=RY M!=_C_YA?T8XXP>C(/W(+^:,)F4"V>>QC@^ ^5HG7BOS8Z1EZQYSY&!I@C@^. MEQ=>') @0!H9&H07FV-.(#KF!'/;](@#%)!EI$W@2#($#PZ?=DZ8+2'D BD[ MVG9#9.DXP6F0D*3,IT)NCIDD+"?TG1,VI"@I.^I[T9_K>#8XC]'?_9@[XHZ1 MI&@P/4J/A^0*$3T*D="?+MDZ3HZI)!19$FR0C.0:.4,R!%"&)7E,:H^I9&4R M.F:/W *.M[X]DZZCB!)>, 1_I.X'#=)[E^-^$$ER"%$D+1E> '>7(W#'+8R3 M461(4$/.D8]('5D&W)'PXR8H9>R1:)TE^4>^AH&D'W/,?3?UGO^82YJ3H@$G MV4N&!!0D<+=.8I$HY ))08:2&>0,^1H6DP_EH^ Z7I&>9"&I0GJ0T9_E^"CP MDM"?0?DHJ).UY!GY4;*0CP+RB$3&DOWD]5A+GI2;S>O8%FAMH@%!V4MBB*(! M0!E,[AJ2.>7T*!.RD2HE5!A%FI+!)/A82.*4O22]MT5JD;VD48DY M4I0S'DQ 1AJ2O23K6$@:D?9D(8E/EI'N)#QI),PM!).C[H@FZ),W MPRUY/YZ39.4,:45"?VVE: Z1I)Q9>=X2+8 S&,AF3K"E9LC'+E7II(M0$CP M1X*1<.5:>53ZE"U -7E8@A/LY VY5?J/)4$'>4F&DK^"(YE(TI4CI&A 2<*5 MD*-FR4*V "(E#4E7A@3#I!,)5\Z5*Z0U259^D,!D8]E5R@'OXU?)-\Z/A5X] M&5X@CR"E(%GR<0OY1;W72TZ2OF7)!T%>EJFD<"D:8)2^PFMH%Z21E:39$$Y: MD(TE],=44I707V;)14:6%N5R:=%%DCCD$3DZ5I?/I$#943*0@F5(R4!>EM ? M)-E2AI @ >>85Q:73.4&>4XZE\,D:1G]>9,KY$&Y/C*-[R1LB4?&C_,D_4@2 MPHU"8D)7)%H-H]_=2.K1)\_?$V ?:">< APP9(01!.3HB-W0COOC LDM:''G MA&0IHDB/C"1>Z4(2E]%E26DFH)=09&C 86:23"0KLED&DRTFR^92FI@@P7X1 M8JJ4F%%KB5".E?>CB))C/H\:IM*G1H8$&^:+Z5KREX:!CNC)@)6?!'@QLB * MW@$J^(RT!.#&-;$I(I47IEV@6VZ6%R9VR66RDM"?4LF*T'N=9%3I7#XXK0CM M" U.CQ>F-4E?LB+O98G98G(8)"5#<#9DF=$?ENE-VI QHQM03\R61P+ 804F9%\",$@>N@?\&(P'-N@)4Q1@ ,(@CT*#O:F*&> MK\!47IB,I"WYX'@VQ"/T)T."!+\C=ZE-RIC1I(^77+*9#$&(J4..F;MC!KEI MZIBE8Y_Y9^J1X(5=(#)&F6,E_W9*GI6>I(@Y3$:40N5U\QKFDF0FG#D]PIBT MXXTS.4J3MJ2(OX9'":U.:[ MF6:&, IG"7DSA)S,9&-)D+1DFDEL1I65I8V#7IZ9ON.,*68. MCS1AB;E5#H\B2KZ)3+(BBF4+:6WNE5ZDBLEI\IFXAY\I;AJ:9YRN0D?>F MEHE6ZG-0QKUY3@J3K,A^84UBG6FFB )M7C?E);798SJ;D65@.3E&G7 EU+EG MFIQ)YZXY3S*=4*:Y.6!^@XAF,W$^S FCG_)7W.F-6<-Z=WQL@HNF7W!EGIVT MY)9Y=LJ4:^:^:5^&F8=GR;EC,I[[YLQYW!!KV"9#X&,VF\4E*WEO-I=(9;SF M4Y9\D>>SR7;*C$MGRMETUI-FIUV0M4F=Q28I"1+@J:I^=8 M65YR;M-CP]ER6I6>)INI4=:>*J9M)T<^GOD>F\E1:AS1)B%9>UJ;9"1@*6:B MC6IFZLF* '=WY?0)?Y:1)^?;:7K&G6=>PCDY@IJ48_!XW?R;5G\8>W$EH M;HJ&9=96$NB6$^7K66I:E,%G"?I2IJ"BY^.900:?R*5%N6^>E"_E[HE*1I>V MII"93%:;)BAV&8+"GO2G\UEPWI\?J*D9;?*2).@1RD%:E&BF>-E0LB*$9#"Y M55*?M*=%.7':=IEDFIAN$H]2Z.O(?F*7)R7U&6J>FF)H$$IZ#J'0)_YI6W*: M"@Z&B5;>EK(C!IIX*BU+:+.9/$*@CZ=S>6MZF=$C-GE?NI!::.89/6J'>:8* MB53^EK>F?@EN"J$H9QKZ@2Z1V.:%&1IDF'TEK1D^4I$+Z ^Y0$Z;<>6AB"0G= M;G 94 YBI=U8=X*):,%\0&CF>\MG(>DZLIX*) 7*82:1KJ9%N7P4H]*D4\5& MGII*Y&=S3MP$YP2.QX$VF41HE'DWSHTF#2JH8$X,\L18Z8J:#1AEPSEK\FO3 MIJM960:?R*;9&7R.GT2FD(E[HI#;PO18?&Z/ALRF6%Z.!"2E \J.AIH"J4!9 M7 J;UB@SVH]NH^/FDTEHVJ)>8$)7-R*($X.S #A\B<)HE$D_PI41CQNIZ>B/ M;^A"\%;Z"N0CE$%MZI]LI>1HDC:C*21;64&NI/UHGMEB=I!Y);6959*DA*5- MZD?"E9@C3&IFHI,A3*"C/=JD[650ZGWRHS^I<[E#0I$])TOJ7-Z5V*A/FGM& M?S-I8NDZ[J!D94IZE49_6>G- .(G,5E>N*19I!8Z:S).W:6'.E8VEDBG>(A M@-F!"@L.*;!9&\(+$>FCD9&:FQPI< <\0I$,9TAJ/JJ6::14BH\"D(#I7'F2 M!H^JY>PHF *,Q:@(>93:I% D#)D?+*;WI3:)DPZE+&E'&J;PI4CI5,IX+J6& MJ5/*@'JEER516I7*!"2E/@>8HJ5IY6KJ>%:4K2E/RII6EJXI5DI(+H^]8UA* MFWJE(B=<:=OMI7.E;ZJ)9KG)V"DH ,:G\5V"#V9B<-/= M!#* ?=I9X*=A(["!"GJG[D(8$-S,%B((1VI8(G-YI6ZY$%RHOX+(>9+* )6H M"+F5ZJ29I8B*F<:DT* LR5G6I"QI\'A7TIF4:2\Y?R)SCZEHBEA^.2=J4JJB M8J@YJF<:0WZ7H"6+BJ("GYQEC8JBX@D )8W:HPZFP2-;":-VIH\(0ZHGT)31 MY_,W.#H+T4*<6!M8"B+ /W@RF $BP*;XB7*6O:.&BG4B<[WC8>$1C,@YJE^JE)R]!1RL D@2JX M,:B*J> D6[E"]J45Y><(I"JF4"JT"$[>E:RJ3EI>(G,C:F9:E5*?C>KI"*MZ MJ#Y>>$BG IH"IJ&I(GH%1,;H]P+^&'%BQ3>L[G]3W520%*R#!)^;X!),$ 1? M@5(7X@%?7:UX&?B)G,(SLEA(&YEB45 AX 'A8_XHHLQU>$ (R47ZFO.=27@Z MX'_!8I>@"%H*$\SE@@)T&=MI_D>L9C+P0L@"+S@-IZ+@^0+D 6>#3*AQ&G,# M:ZB)7[R. $.2FL5:99<'7*(@*L/:KBXC?")BL!+T,/9$"\ :5!LO08[B M%5"&12

FK4MK MV^JW$G9V0.#:M*:M,VNK@+4>KE%KXAJT>JV-J[ZPJU0*JMB%QRM M@HS-B+XBHV>%\CK>,:\HC/SZOM*OW,;."KBVKS:C A :D*\,BOF*OKZ<]>O. M>K\"L()>G5>S"JZTGNWZO\:O#6P8@;ZZ!;0>\(JUXJ_AS /KN^:O%6RA>> 1 M%:G%1'H)-A60('#A$_@%Q!\=<$DP""C!);%I6 1M0$V0!M!U&<&VFAC0=?9! M8D#\L0%UQ!J1$:RL=80OL49\K,R@5) Z@!Q)*K:3' JM/J.WF.\1HF9#-0G% M^A/M(FMH30HK_ 44NQ*TBTXL[?D*_15>[!)K-F"A*6I#<+7B#E*L-/JQS 5/ M+,*:A!0E&0<<"SPZD5?K@0#&@I#I28]YM8HB6>R[*3[B 'DL'0NG=9'.) TP M=XJGR"G=-Q:,?B1L/7&EF,6;(?F=%(T&[!*,Y"I<1!):( MEJIA& 80?O2C$AOJJ;)X10UQ4N 5.!YY&J*,CF=#+.MTU*,?;#\JRY:@)0>. MEX2D#JULU(I7F(N^+!M+S$*OQND\:1(*-LDI*AC)Z@1(0+=:R=X!]*2@M\KJ ML<'L&P'+^G@F(1)T3M2RW*S9P,?FLC->-RO(@K/]:&\#S,H?-8@VR\X>L^FL M'@O/+J+)K+&WS-H,OJ8:"LFBL//!8'B7AA&I+!N[/XZQ\R4JZG3HI:Z73X8>>E@VM-%J%LH8PI$5;2[J>!VUPV5YFM!.M=-EA.K2S M1&.ISK:JUFPT2D[>EQ>JV6!*AIF$Y+)02_ZF5*EX^5NREP#"ZW@LUI*6*=+G M4D*/\R7F1CM&L9ZH$[N#SK*T9[X7T8IOQ*,J:T.^EH^BG%AZ(J=5*M[WS$H! MT:P\@REJBMOE6%@-GAMG86]3E'AY)RV.!YZD#BKMC)G3!K4O[1D[.LJT-J1) MB#4>)U!MG%@H#J%4K1JJ5-XX6EP-<2]\M69A5.#,HK ZK-/PSP84X$:H-\>V MJ@M!T&D]!I<.IVH)13XX&6!14ERBL:VHFGG(JI;.9$D@UI:1,FTTVM-:HQ>M M. FKU7L[+6"I@LYPOF5C>X)ZLVX!47MDMH^P): XU8H=5:VE*,E&LWZ"'. G M'GH"K3ZYV.)X>H(1F=M>M;SM1?#;GK1D[1=+W-H,QJT^^_P]LSL*]+$&Q'V? M25(@.< +[.IJA[)NMVA!-W'@5;=""N6@BQ!\>PAI)YZFLM=)W=?'E(YD*?0' MWKH*;0AK2*<)E/.ML4!^)I/.))21W[8AT&-)NV/RDXSD?QL45); &_%HX*:0 M5>C]Z-\2?,9"]JF(YIZ](PBPX,:5DJJ%"^$"N&3DJYD3++C0W[M9G"J=:"AK MB..HH0VJ9C$VDG;=XR&)3CXX*^M\NXPD@O6$E1K-6@3/RD(0$K"73VB,2_#- MN-9M?8CW7:EY*ADP%9B.L^K3.6W*#P#$CTO>UKA#;C3;LM*P?213F9_2CM\M MP??D"KG0;,G:1X8$J.L3FN7""UON\X?5N@K^@Y&KG(*3S:79Z9WX$O-MF8OW MG;E3KFXPV%Z*6D:4^5O.ESDM!ON&CJ0^;14J8N*TV^>K2>A"E^"G%MI+WH^" M;E[)DV*;9.5=.::H21N(TIN2K=6+0I[YFJU 6N%ZJAAFS5;O2=(NHFX[N099B*> M+BVN"^PZL08H"IGKSI?.917*ZTJZ]6B8*4[JN*-C*IGK,KNC8S0ZT4*[0:UI M6U*&FK=N4#MYCKANI[%G9F@T(JR&)^?!'W*?9Z ]YHP[8Y3)7."X4B64$2/4 MAT!;=ULA2 $MZ\WP2'JMCIZQ4>3]IPV"G)HTR@'N9*NH+%8()PD.D=ZFNZ$> M%))>G(R:BL^Z;B2\?LLUOSJKPD[]%1IZ&\4@#2F^\ED38O"C %2+U[&LH+ M]#:]3^?FR/.&O!POR]L^@)9?;]'K%W*]OP*,:_.B-D_ RIOO=JCX9M5K!+B] M[,JAJO2B=2DOW?L2Y'L]9]5+!.B]Z4G9Z_-NO?ANW1O-H;PXKTZ0[X:X=R\* M( 0 OE)EXQOS*K[LRJ0&]4*^:D[C>P1 OE(G+KO[@9JYT:SJ>V6VH,CKIO+2JL>JO$@0DK*GZK\D$/"["&JU*?93@" M?A91PFHP98 (@!&* +4!!1@+EAI'@AW1(E(5!P&?)P"\!&^A-TBY9!8&I;; MY1O!CM".9*L^.?^.CO4O4-%!8I1]*V.+FMIN]=X;P3M>E@'PKQ!1]K_A[4=J M'CXX!K"#,SI&E 9PE#8Y;I7'1 1:ZKZGYT0%W)'2::F'%WD<)L!EPHXK+-B_ M(J:;V0/>CAAH -Q,>I/OI@W!.W*: 3"I]CK&P"=%POENUL!JSO084TFP0%D#.\%-8M3:04K!J4C^NY0:JL-* MK2F4AH\2\%**49ZBO&S[\UE2D7NI"?QJ H^D@0:*0DZZFRFG"0W:ESIPA]D# M!Z5[<'1I!%>@'&4AF5GRCN(E%7P%&\+'G"8<7=".83"2*0=$M7>M'@D"XWIG M\+V(9[AZ>-_T2RH6MELMA6FNMI*I9AMLZ9IQH8$2.0OSCG8F,@?8JJJ+RC":Q_[J7QJ""[&[VF4:50&L+ !2SDZ6B>T^4%3NI#.J=6@N&F+]K.20&9M]TP!D("Z'='A!*&@L+S@![4N#$Q5Q]L>+:>45+4""=]@;= MXBA(W%1\6[&\$&7!.IX,8^GFY<&X?"M"W2*&X& MJ_U+XN%'_!KF+4BX\*8DD: ;,G^TJWC?KE@4H(B!(NG)+H*#0&(*2]&)Q=YB M<+CD%7/EA77X#DJ,JS#"BA CO^UP2.@5_W[^H7:L7,2_OFHU"![KFL4<]+?, MS'4'L=*I'CL37_$SZ!Y;"O!QVUD?=\?L<)6@"OL9!]YKJ1"'A-AQI?@?<\5@ M1GCLUPW(X7%;2AXCR!?KX8>+/AK@*.?R6L:6CP;ZIR4"B6%Q@(PD6HF&17QL M(-_'%^L0T!2"@:#@&M -A@L50GX\\"[(JNLYF!V7R&#&B%PI\LCRP4Y\, [) M&J&$?!90R#;RA1PX\'4FH=VR(C*#Y_'TVBGZR.+@@\P=A\>.77T<'LN, S*+ M/.0UR= O6C 9L+\LHA@LU8K(\N"/W!P'R(.BG)@B%\COQ(&\)&?(OZ@0<'S4 MR**PDJF,](!4,I\()(?'K-YC,!_#R4IR]G>Q3@:7(F$@L'[(MBV?G"97R7_R MF @H#LH6LIQL*'_))8!I>)'4 73(:Y 1- %9'^3']I5]5!_:QQOL@T$ 9F Q M/,@ZP.CW I@!=H$"DB%,!"#"16 XB!^A\M;7,.@6J\7H%RU^#.#Q;K#^>8LH M0/TP!^#*:Q_7]]6A !*BU[BGCC/R0]4 ,K0/P#*.80>>!;[RP2A 3 6SA80R M+%<:$F.AG"UOA,)R/2$P@,K)\JY\WI4>K_)K(/LARZ(RUV>6T!E<0IR<63PI MV/*GR"ENRQ0!<.$MUQ/5,I'B+0;,$V.7,"Y[BQ:"V5?UV8+2B']'@[3+?:,' M0@LNS%??BW O@HPFH8X8!L@31<$5@#J@!5MQJ_PJQ\JOP:P,?> 3MS*Z'"_O MRG,)^_$K]R3!\H2R!Q3+K\&QK#+KRF' LMPLPQO3,;3,,VLJU?*]? M$\RK<-.<,)?*#'/4C-Y1,!"S5R Q*\RF,J- A%S,3#+>=^YB!FZ#7>"RF<2? M6M"@5N@@AZ /PH,$:OE!"9=-L8XE@2,;\+*-ZL?;6.1!?V'Q>MPE2!L1&"FSG4G3@P[;\AVY]X),'JG ./SAQB( 77 &: BC _:24:@-<(AM0*A8KS?:>@,%.RK &,DFSM'QVM![_P[(Z7"\]Z2-4R1F,CR&-NV MMW;*6/+,,L^;(OG,^"Z_YRK^R>6.SJ6B8?M@1IGD;N+1V&K&#N3@W! 4SN!S MP#MWFH10@$<7_UF#(X(Q6!,TR.)@3> XGP4U09^L'DQU]T-H@2UPOQ*J@G & M=G\B -N("SH:"LH:\6N$QPC?L@P=<3:!E1SRA[ M2-#WMLRE!6O&$( $) &@<@H0MXD!1F\*D%BB""A )*A&VP5L]''C9?R\H6%S M ^9M*@!)W,*4"; T5Y!0J<#I "2&EM@!*@!*( 9C49# M 8:T6\!&+P%4 "6]R9F\9<8?'4AST71 "H &H ")=,C'2-/1:0$D+4FGT4() M&WWT!I9PM!Q=)AQZ*( =;0;@T !(1K(TP' 2G +2T$@!B.00J &='29@1'!WQ3 PI@(C"1I:GE 4NU1 M,]6 0TC]5&_3)35JU0U'1U6 Q!*=8#*&S0,#TP*< :$U7B!(1W-L=$Z M-5;M44L*' ືU1##KG8&.-:HS5H=5V/4X/1M9UC])L!2[/1T(?&5]"Q 94T+7T26!NHB!825V?71]S[FD\' < % M&J >"--OQIHA(;X!<;17MV8 V()C"N &W*N0 5@ 6P\!]9T.0E_3<6Q&!8%@ M[]>@X&E0:1('FS47D/)6=5Q X2A,)P&\P1WM%ZX9SL9G_5C;V#, ?CU*=P=D MX5F08TO4>@A*<,CU!1E!WA:H"0N9W$=YQG+7C_2-#7,YTYET-#U)4]/6-#:= M FC3W+3*.UG3U;]U518@U-$2M94=K_%IV$X-XBADUU-0$&$$C-(+]AD"&XS3 MI1IH$*F]@*HW5@ M4&:PT;^U!.?@0!E_-"2]'S#2!+75FV:LL)(U'!T$]!-^05S]6$,!/$U)]89]KV* M::8 J1N]]FN[VP*WM:UO']S"M@)0<-_; [?!77++VQZWR9URH]S%-FJC-------------------------------Cut-Here-------------------------------< ---------------------------------------------------------------------- / e||) Lyndon J Clarke Edinburgh Parallel Computing Centre e||) \ \ c||c Tel: 031 650 5021 Email: lyndon@epcc.edinburgh.ac.uk c||c / ---------------------------------------------------------------------- From owner-mpi-comm@CS.UTK.EDU Thu Feb 25 12:19:51 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA26012; Thu, 25 Feb 93 12:19:51 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA06370; Thu, 25 Feb 93 12:18:21 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Thu, 25 Feb 1993 12:18:20 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from daedalus.epcc.ed.ac.uk by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA06357; Thu, 25 Feb 93 12:18:10 -0500 From: lyndon@epcc.ed.ac.uk Date: Thu, 25 Feb 93 17:17:58 GMT Message-Id: <2160.9302251717@subnode.epcc.ed.ac.uk> To: mpi-comm@cs.utk.edu Subject: third part of document >-------------------------------Cut-Here-------------------------------< M3.MBW=4X7R!M"=FS[?T0"!P-9;F G&V9^JE.Y=8!$JUT,N0>,&HSGU:"W Q? M74UY1V+0@-I&:2=3'@V7AD_#)65_,*=O/LPF1:NN;XC-4J5D)^?8$M:NA3:K M"JU#[-7^+U+W_SMD+6*\V'R6?,_EX%(7H]==I.4%TIZ]]\L%WYZPI@PL;"Y# M\6[*!V29\*AQJXCLI18RE?UQ_\_4(;KYB%Q()ND9^C!Z,E7XW-2X+"<&H!H\ MAR/*J5^U0UQE3F-M5LCVZ?)IC.2I[.!4M4IV35S8F'>%K>)XH6Y8(E8X?WL[-+F'>J/_4/4R>)/\,.!6NS.9[:G]!9O.JD2DG MG&M*B9N:\W[V/SP 5 4+B).>B[H4,]%GT;BA2Q"#)PVG!F0T[.//ES MD$N_#>7*7^Z60WQ?E=\!]3H$N=;M*764C\8AF((8<4<# M:2W%D+3K[E.21LS=M1%G.*-VQMUU[IVW@ERS=)YG\PW%""J0A2XX]UCJYJX4AN*:Q-+E]]V*$8E87R73MPOSM_^ MB_&[>F)[,C:P3B.LF()*8I1;K#-P>?KVISV)/9+18UENXJ20J4ZW%<=_MJ+88R\8MKMQZ MB[EZR^=.7K$WW%QOQM!9 65U.%UGKRHYVIM$(-U6C(LX\D?*+R#UK"D-?1<# M3CNDW=YY<<$XEOKF0-Z2>Y..G.R,:?,6Z61,E%+1EO?$_&1T*7+:U19;PU9-\>H%_&#\R8\=&W!UVZLQ'7 MC&&^*<44'^ M,%>$,&@@J:L2F+H/36+"G^?&1%R[,;9CRG8BUKZF$*_/>^-5#1+5\-I]%AS[ M?8O%Z-=EI?BY(IK%_?:]7YO%043'\G)$(;9KIHE#1] O[/?Y" M.\.;U6./K+S%ZY(KR4^4V8<._D33RK_DW,)?^!7AV MCU?1;LU+HL9U(+J-V^4&5.@-/14&!\,^O1P]L8QK?Z/BVT?(&SJGSR*9Q:I3VJ M[\BJYM6+85Q.E0P##0UB&+N6V(8F\K+-]A=R;"2,'%^9XDS)J9!QD28@P@M4 M>56_: NO] '6") $>)3"%>FU_T_9Y8J79HNRJPYO=8ZQ #[4!FXXTNK9E0J# MI/=[_+$:(,OW?EMDS:EF9B>4ES=R,$[:]UF"/"1_Y:2 _\9ZE1Z8ENPKQ0XS MAVO2F$\'J;?6++L?@J>V!6* )>9A+[Q4BAS3O3>^6_D#7M_G87T3-#VQA:.B M#M=LVUM"]-Q9O9S:>TMGEV5\N&&08:.. -KIRS]OUO+2)\8.](0X+[>M^Q8N M"@?3C\)KI@E2@N:9MMVP.=:M8&F9D/10J#@:O@($ >*'.1QS+VP.L;R65B][ M13]T5N2:[F)!=??_PR\3**1_M;VHG>W/(IVMW;W>+],$0F;+=-OD6?CI ZOA MIB>?Q6-)(T3Y'(R3'B+W[9S.FZ-S%@T_ M 9S3\4.:\ALY)-R'.P>CA57!6+Y(F^XUP<<&XB\?>9=TJ^(&8,:9[:?9PU^" M/Z&30N9DW7V3SG<];0L[A?/3(-2[].%/]5>@-C0J6F'%"[[>G\L0;@#9#9J. MEP>A]&DV)6AR5#=Q!NXAAM>'P<0R,F NU3<<;;!FIC]])Q_TE+DZ]&&!.M'ZNR[F#\*@P S^KZU#:G MD:5W+KL%'DMX8?CF6>PAJS3,H,TPP1D ]\!BML@Z$J8#5]JX2EYM--SRX[4] M>^_+3]HX]7_:Q/RA)>H6"=/-M,M:GU* U,9K3+2:[X ,8]>38\KQU"Q@,=%5 MI0V&%S8[\ \ULJ,#K2:J4TV:U&?,33&5B1JL/6DZY'#2SDL4@:AR$+II;!T* MJ1O+H,MS7A[U27P&?$=W+EMOBIPQ01TU*]W2:U3_IJ74Q] ='C&RVBHP63U2 M\V"J56('Z3WA]7C,Y)Q"I>=XIYP'K\.VDS8C;=6&B>&M%+\)M;:3WRN.3JR] M" .:1%ZYW(8U_\PI[D^#>8"U643D=,;5)\CM:?VJ $W31%>XWM%UM?:P3K3> MI2O6NMDI[YU94-S?/=?6;+:!PL#F]'/ZS?#P-7UVF/V,W>HSX[IPMH?7,\:% MY7(,-]2"()LCEK-$!2(.K5F]?%>\,1(7$XWW%3S;>E4UTM3O>*I13['#C&#O/GR>WOMZHYMFQ]39^"/^M M54-QB & @5FU+;HCY@SK/7LV+,$ K\+Z^-IZPX.Z M][S5E.FO[I>:+,>WIA@;KY);VW0C,=-F\>:S9#S,[-AH#&W-[XJL7U@M7Q?/EE3 M--ESD#V6,>J:L B%CEFWKL&1XDC?9V:RGE!>!2/4F9ELW!SV+:H0!>@?[ER_ M&7R??=8:=>EZ_];C4S>O;XO&Z&)/( ,78?VQYH42 5K7L==0Z!5U<6V(F[=) M5P_)7^,09O;:VO2I"6#N*9%Q@]24<"$5 MU4O#@I< F=.9R/(KM884P_9!C; M7JN9IU"(,<$W>JME45];-AX8YA?UJEE_.<^3F>(QZJQ:,[U%?= MJ''!$.BM(>OD?MUYG6&'?1&=\.OLH8[TL)LV3E7B< '8;>-%]) 4;LP4'5O; M:PR'61/JLR4ZN<>T=C[>;*^AZ.[,=O;1I7+@+79/N"]>GPH^VG\7' .F8C_]UB$-/59.6 M?=FD!=C,L9TS#72+[N32<7FPP&,RXAYQQ?:?[@569>XWRF@JXNN7C]V!'0PJ M?P&0A]R=*$-'']W[?>3V_EBPP6/@L> T.C"#3?X22FEIS%^4WYK0>2R$1?G] M8*>_TV-7A'FC,BR/YF/GH\N_605^M%#TO=E!--C>;^38[]]D*,,SYUC_/4C_ M_9C5^M]@+K3TI4L!M,#VJQ-W;%C2-:V:E[DJ_N!N/R^-VF?D&T_-N_=%I0P! M:2O.R@)E*4=3]P=D_2KZ_CS*C^8[M7UWGAQ/[B6_>R%I8E2NL,]5,V?CBT,U=@LCW8&JQ"CBAW M8E7!ZF N=#Z-;]I@A.V.B!4GYXTC*E1(;ISI_(_F7UW$9^YV M=XF[X,S[;(*CH\@2O!M229O$\D?'-? M7BF\+&MW("X4K3VQG45^7/-I?#Z4W4V3"ENW"2'B K^21MPE-J\8^]RF"1"\ M:5:!4^SR:Q>3BAOX/1Q;K>MH[E0^ MH]&(Y^C:L0@V+#C^A6774@-SC-S ')#/.=?[Z^."U8S'@MP_(A"VX'G\?67O MHX&BLVQVM"-[VHG]W5@&MTNYTU]9;A+3P:$#M$Q;(),^H-*IZ"BR)/!(VBQ: MMU62<]^-&N37C&L2G?M",B_;-^@^'-O.?N!,C52*GW7(LQT=L %B M(N:^%S6O RD-W]@K/1V@ME?'J^TOXF_;M0W>U.+("1_9LFU$;L^OMNT9NFU; M):N2CJ^+7V!.-;@U@&+'*+>Y_ MN-?1X$WGY7+;".N$O6D.K1_;I,4;=[$4GYS37."-*W/)6%>)HS0%,0%4.XQ=W7*S M(^S#)6V$(I@[G[8W<,HU"?%S"N63F9HU3DN^MC%WE[O#TUMX''UJG'^$G[<0GH(=?/5JJ^;X\6X9;HGF9KQ=8CAYB M6I/.9FKE7><.NRHEY#3C5W&R5J)Y6XO.)P:7)=CM^ 1YF(33H:YNTDLY7;2^ M92F@L&7LW4HFZRPB?CEZY*Z.T6KUY/@T6QP30%O(,7O/2=LOIR^GD/:TJU)+ M:,O(A>1-'YP8Q/?%0VWTCP'(/N1,JO;X:/L$SGJW >5WB.(]HC'4U?@%KN)I MO!]K+P VIS:X8V@O*F=[+FY3X4AQ!S%=_;5>X1FHRK.9U[6T[%#7P M=I>WJV)I8<:Y>GV"G"QZI&>V$F*ZJ_!&2RF2)/(9<3L$#L_";5!SJBUX=>J5 M.7RM(A6QPU?24 ++9Z=_0R9B/)0WTW MMH@-O%,1=7C2D^OMT$ M!MV_<,SI9@!VLEW'-(E:MO?8_F[UMFC[CUJ.'G*'MC7;A[19]Q\R#92!E0*J M !BP?MP_*2;;=4QZ4P'D<57'5.ZS)*Y[V7WE/DLVNP>YP^U":20;][OF+O_^ M_.9>'+8G0'M3_!.8 QYC^C2_W49*;NI8*TS(WG4?CS6Y>VZ$HY\;EDWM'G$; M$F]KS]]0=F P'LW(L;X0E^O7[G MKF^SI=:TO]FMQ7A0X M*#^K?MT.9K)ARO?0I%G:+XO/"(<<-7?; GQ>+B277(/5@0F\;J8PHNK0=GI7 M% F'6<>D3YBV-' )S6P,0?/#66_,P5VW]>CU+IPRP;/9!%3D\XJ6+FJ;ME%* M1\VTB%48L0=2_PQ':W.&00W.D^4D-)Q62ZT!_()'2X?*06(6]A891RU08P@* M?9&C/>H*WFURM?9)QBJ7'!]K3H#[&A.@3UJGHWQB^0+5W6R\G/E20GD&7#P? M6\&>"N'Y 1) Q _1.VID/AO%!?="D\*?RA&U6GPI6'2)"J*P!7$/[ [I\: MPH>7B/!YF^V/GKC+0Y>&L.-U%L?;G?_81@P,/Q<2XH;AP;E_AF,)".%+7B1P.)\M^ MY3 MPI&FIW"!^*!:=9DTI/96BA'B$;UON 6O(8[W)"F\G%.\@CR/L]]S(UO.W2V' MAWNZ]G_4RLG55FO[%+GB0NYV<;97?JW6[+JR-7&N74,_-/O0VBP,=&N# M0G_:8LX?KMO#<9O55@CZ4Y38P6\F-O/QM!B]$;'@:;1J@6TI+K&8L'TL=GYC M<7MRESAFL:U2_LI$@JX%U_AP^43W'B!.0)G@$_9R]MS89^3WUVF %7*;^2Z_2\3,+^D8=3S_WNV]\O*X/VN46C]OZ,#_.TM^ M!<0 [6_9W73@* FX"8 GN*G1/3]%KG7@_JC;SC1_8@'9(+;[-P7<*>@K!;%A MA%"_NW&2*@:<=NPIU/RJ5-=KSU],MDKUE"T]=L$.8>W1\FAM=XZ-JEH:"&V# M?WVEK5SR.-"IT"V/AHFNH\GCL0A)WO-W#*!.0SINQL4.G?%"X,PY\2N+2"*$ M-.NNT4V5I&8&WCG>[G2W20?D>T<:K^47-9[P4XW_L7FWQM_7^.FXX#VBW-'1 MQH5IL%^$ #N3M;W'FPY\H^O;1;_[0@\)IPVO94:'?VFB"0[:[_\1MS:-)H!# M 7#'O-_Y5''<&\WY#61/P /<[@7GN*I OTH?'T?#J&6_XDY+)X QF>23#^%G_NPU9T:'QT_ MR#'9@NS4L7]PY'VZNY"'N6_C$?+87'3.[X\666' !R MV#2_F#G@KW*\=6PW=1#H5Y_CDKDUN:,;NDDD5Y+7-YW:M>?W*(1-0"3?='=& MQ:#DG&XIN4GT45XEIR(NR+'D?L!DSI8\=-PEKXZOQC%^R/'-[R!;ZDPFMY!S M_S#D:/+X-Y]G3<[M#KZ>H]_DMU^+*HQ<=YR5M),/TR"Y>7+@L8+WX\:G7P4'NG&_6;M!N=$FV29*G#LW0,-CIUM _9UO/&W_EF7C9?)3^9E<0ZXJ[Y&3 MN;4X*G(V(JP\V$V ''8[N+$,[\6AZ'8.R9>P/(XOQV]U+ @=^9B 1RXLQZ$V MRPOES/*]J^%P(0B^S*Z!!B*0A7)N-]AY6" :)V]3RR?FH>XM]\28O##,HN8()=[N'M_%/+9N+H(%XN30:3EXOUT?K?@-S5H!\.:V49D8CK^16P'WES]%AXE8.9FXZ?G82 M0A_F3')CN4V <[.X79B'-B3:9,<60NO;77?G,]T9O.,K37QRV+#IT?8,OF=^8$=N77U'WJKM6, M:TWG$O;)_-SN5,N70Y]&/ZUS'/C'?+)N1_Q2ZLSAV3' MRGOFY=^?.8..\3XX'0)E'SH/EPG.R;7PS0 Z5.ZIH MSD_$+)N?*L23W1VS>:V^N]]T3TMY]TJ/WLWNV\/F,Q;"/.N]MR-A-_D$M@.2 M#U/ W6DQ*M>SF_M>KDLW=.^*^&P.L>3S+7>>Y#K#S26N9(0(8+T6YET/GRV3 M5P&?"MVM6%D84HV:NXW39 >A>[R)LYZVFZN_>\S5J^"R0\\-[ P!-S??742@ M#:.TI9K^ IK.ZZ'-5/1YPPU'7RM9:$_ MBM%^35V\7IC [N<_S WOIW=I2)\ ,[^42^>4':72XBKHC6F0[$_\3XA 59Q: M!JS9?6]5)@CP%XO0QJ!;&R. ML)%H=(WGT82ER('E@,USF;8Z@B[:E@7WDBG M09NZCSF,4+!/5QG\9O;=?9V/YKV/@GG/DA88O]MB2:W8S6\L-ONU^ :M/$4O MQBGFGD_WGCMYD!>,:!@$=2C56KQVMEB4C#Y%QVHZ$TF0O 5J*?DN+#<@2!Z> M5I_ H[];GLB3.*Q\IKQIP:])*C_;W0WT*%H&[Y[^#".)-&XI3F P47-S;&#: M! T"N9Y4V[Y<*PGC:+U)"MIX0-[3UMC+G>9B\X*55UJ-M2H+W$T9>';:RM>17QO!QO%M\ ;AQK5@LX-I["! M0;O5;DIJNAY:Q1RJ53BJ+G,XX_3&M DUJL>M/ER[*2ETUFPQ7=293LW1M4/C M@L&@,Y)%BAJ50>KVKJ<_GBNX8X*Q76M.K+K/G@!2!&B9*<_[&WRW/-R8#JA; M7,O5ZC1N>+IZ>2=6)H":J!$")665J;X<1AQ3"OENS5,O9]S'7]KVQPWUG M1H/62\K\:>62RWLWSHOOM3/1CIN:C6F$*MB)5GX/QIG?HFC#>!8;T[8L-J0/ M;AGCG#@QMB>N8>C=LXR[T_&OM@9R.,?0JNRL?1K&U=7>2\LJ[[6PF8 MYY[1:@Y/-G4\<&X['L&68"&EMG+$*H?MXPI"UM$Y,RY6. MLFNPGG(?[/J-E(W*566#QTO@*O!T$Q,V/.X"WW9#,OM:U>/)7L+/%5%8B:[K MUOV(^075^AI15[=7UWS'E CK=9I^WXN;QAW JUQN:DJEE=(4.<6\0#XM7RZI M)TWC[_/HHB4,44GCY9(WR+WDDW4(N:A\ I[:KG(ONS?KZYHG\1<1>YX;%ZT/ MS/VXI[R].IY;U'A2G9>WUL."PO$I *V\3EX^<K%Y86<1^Q%1_1;U^YJWUV5"RSIXJ[39U U9OZ][ MRK_D$G#/.FQ\/@7@)OT"V'ZY[R)XTW[#O MU?6K)W)"J$*MW?ET'$@K4W,;GL MY7D;O>ZJB=1\;_DU&#)6'R97C_?71[[*[5@-8\ 6:Z(3L,//0NFX]39Y@-ZVC M' ?=^6L]]VH=V/U@U[)SV2?L0]'8.JYI0\S_Y1R'1V%,GFX6^.>0F'6^XOF)UC M"T3FF$-F[;A\O[W^=KM=UOWKEES4,:K]R XH9ZBB6L?DEG8R-T[ I'KF9JW/ MMC>!A'/<]M7++.DG:(#K@+Z>O'+B\6==8+Y;'R.*$H'@DS\SZ M6A-$47.0:!MUTVW'-I![O]NH&7,0.9%[Z7TVCZ^[SINHF[38N=OQNVQ\[_=HU+SBGMDW8B.W<]V^XW%SV^RM'

);9_=V/Y^HCEG>9?MI>VRK\0=TT=QC[:# MRM?M*C7 @D=1$^=9Y["CCMWM2O-"=M$/!CBWD>*X#TWDSNCAN3S\X^YM5W"_ MR+?L5@!(Z9M'$MP 7[GKQL?1J78SNZ4=S9YV)]ORW%7L(LG#Z^=;*.%K57#X MM6_M>9N_(=!=L5[9_AL&W)7)>_X[LYX;)[ CV#/NH?67^[L=CLG YQSVK_M!? F@/+ZH7*GF.^^*,F+V!V<]Q$ MK0?;<'7B)I?T,(YI4QPS=:;?7FSRL2\;*H??"V:#L+<$.5G>'8%3]YT$1^I1 M"ZGI4R'G7.X/['Y,W$QSB)GH._&4U]'.U'=)7TS3>4^F>D]WWYROKP=) [M; M/@_%?CHFZV-TZ(Q8JT^^HR^&1LHRY[)MT\=^OUTWO%FLAK6=M,5S">@Z%1/. MW]U]^+U!,6=2^M>3;4TS7>&[R^:8^A(PA9L/+WH3_U3JU3FQK-OSH2[76[7[ MA'UT]%W2VW:.9"E/I(UVU^W3K9NT..22>[IGHZZ)5*KF1'6]S3_TYIC#,9\Z MY'#IHH8HA!LP]>CE_*5K;D6\Y3N6'3$=0\>@V[\%U-&-RG1/[JB8GGXH=O02 M.*G.QMRH;4XY!9SW-GHW6M^6JLO\@M>W1\[;U:9W;3^]H]XS@+Q8KA3G)*C7 MJ^':2T CI#B3!9^8E<,'TV8DFIDZ?'<];)B!MGKEX:M\XDS_^\!0!5]+[*2; MP3N(. $=*CF-;>/4YJVFGP&OTN)M'7L[$1JET0&NODTXZ;>#8+3OJDZARZHC M/[?J(DJONNY[EMN)EVA':=:VQ_>"Z!T=6(DH8A[6XK]I??1A\1]=;QM(G[[/ MU8MOY6?I]_DY^Q[OV\3MXIYUO;C$P!SG,5!9(, M;R#;*DF>I(MX=.YO_[9IXX728T34L7+YH;O4D%R (4(,V" ;F%91ZW7V>.)ZE^)7WC8<1%3Z44P_O\3:[7 M#:WK>>LDIC5^)3E>N"3ZV1B>#ASM.P^Y>BT?!UT7NHB9;\\,[JEXIAGQXE@%@G_XICQPC@9-Y$31\$R!0MH M_B)Y1<;EW=-R7.B?^\E.&_VG(C6J)[<7AA<#!@&&&IJ"_S1\;!>:C[E/O&6. MO2EI\/>GY)RP#7Q40M/=:3N&/^(=\:86\=)D'3S<9Y95 =X+OKM:=;RRVN\H<:V8*GR0@O$"M]UG6F<_#J9@C=7ZRRL M/K=[2NK"ZHHW&NO.W5EWD-F&'K@>+59S,9JP95O2E1N?=^5,88!9S0AT_-*9 M6]N>3NHP:%QX=RLW5(50U,C,.V(-AZ8,H]6@V:V]-+ADE97:ULW23=8]?,- M&2,>34L9Y^J44H@"+2U3$':[4+[=:?@+.]JCI$9_.[K)-40!N^"( !!-V _1;&H9 "H"7&Z6I M19FS(U[>PK249CLR'3DGM1UVQ\S2Z>EY+SV6LWESV!Y-#0*XJ@R/7U@$_WI. M&M6<:UT:?><'..@/8.0RW'2ZU^ MUN9J.&\P^J!X>)GIJ['%T1)"9,9V,M@63;T4>%+P$\.R=P C9GW>7G>?5]C= MDY.&'&HRH,!;!3@%H/F]WD2,/$/ WO-PJ"JD 2 Z#[N"0Z)TL#/\Z.VI\VV9F46 MU\J<,W3U'.7OQBP[#+YM-A2X].ICZ_YT">I1*\$C4-_'YE7-W:TI=.9A'O0&OK]PK&NS_I9 M"6EIMK]W/9S.*DNEOX@G2./5W=70.!]U>6B!!J5^ R>8%-S'9!(Y:/SEH?>Z M4GEX:O%K4K<>9JQ [-?KB]G$Z]V?,?#V-FVQYX=^ZQ&OUM?R>0Z0$DU'E\4W ML6GQA9HA7Z&&H):+'QS_?4/1TO( 7:E>T)ZB"D3G9C6RVNSP^Y'YZK/D5 M 9@ "_"4>QN@D\UV/T87!$<"9=)"X$A@J#Z%I8KVLB?RV#LO!_I80:\^MNK& ME96-<;6!<\94;U=<[;&UYF;7U>4B)?A6T*Q=AL8VO+^#=M5D>*C->%H)F^@:,^5C=KU MR[/2@;F67G2.PV9H=0/Z(=U[;F?4N0[N(ROFKP,QTU;?O[947F'6D;Y<5JE_<2:MB738%';M$5/QOUHO[P= MA9US -DA MJ6-\NU5@]@Y^Z[QRO([> ,/.HX$Q;?H->UFN_FQGO8?UPE XO+'1[%N&54E3 MDC'?9-@-O=Y;@;QD1+ YNK?E1,U#8K+<44/_35DR.86:GGJWYHD=O_"6)-5[ M'8\J#D\1*0[_ .L6SV%_.7FRQ%XX&E?8(V\B-![8+\N*Q3RU>/9.KBE4K&TS M3?F]?G5^X&H-RY>[K^%)%FO4"[Z_(M0F!YJQOW:32,/:[^L&-%D<7NS1-)+2 M\"^7Q$?\@E,\^XI\G\6?Y$(#_,M/30GG^;[\GEKWXG7V@W2G2L^^B\W43OQ& M$A^#(%'M-U(0K294PPL:.,RX-,GOMW9S3'@1(*C>Z* %C7Q!J<.N42\$B$K> M_#R5E;2_9,^/@9V M+&!>=""]$5F&05/K PU/ .8HX&P%%Z)X8&@;$BQ1--= M+&U^9$F-)5^0E ^D1#K^\466:OO ^B419:GN7C!N!N?V,DY3YN049RF>W%E. MW-S3:_1EJXI.U"NB3R_<)^O@N6&QEOP H5<^-BLOB&,1R,$-MJFO8WC!QOX8 M,PNKR"I-Y/&TE=Z#_QE>$UWQ7RF@8.!P.YK@:.$[Y!CY:+7TJ0PQF5^=E[J? M":?(U. Q_'*TPPB'O$8*(4/CI7PC*"LP 2EB,_C@3#>->F@N=6>.4L?@Y:T3 M^%CQZ7QT/AKS*6Y1S-;?1QV'RNF&N8G><4O2M%&:%,^ 7U"_%2.FL?0]\)# M&NGYP>, \$VS%^**K]E(HIND#T&W[5X_S%]BV.+LDI9=H[?LW.WDV/NX'\X#OYO6R?6OO8*KH_YB2_"E#)3R]> M\E5>?<&PH $RZ.?9#LRMW@_KT_$UHK+.GJ_:O&0N[9^BO#FVO9O-;?_PM(IB M[[*BXNN('F 2%UFU6S G336.SL'0KF./!$U=G<>B&1_8_TKB\^Y#>[ MYJ3)I^88([CMU/KR[%UJ(]1T]^-JKL]T-:Q7%#D>##F>['LF'\O&R2>:%&Q& MTE:\#M8[&F[4XC@_1_"Z&2%VM[S4X G P]@=W%G:_@##"6"9=I^1:M<<)''B MD;.-'TK#KA760@B;@Z3M57=]D],7(0N>4!UA]NX%[+7S/^-J)EOYLVO8;P0; ML\V3OL(.\.\O.@=-].P/325$#CD&??AOO?,$GP,3#*W2EWT9868?Y=CDNX$" M4?-J9L;;7G;V?4H7RH7_;U5TLV&3\OO4?\"5AB,I (W&F:\2X &V40]S*0), M :Y>#H[E[8< ./!9"W,6\?WGGV5W[KN5N\%2Q>^+-AC\$ULCP'.:"5 %. 4R MTM#2IJ$!_]6PN+8EJ!HKG^JL=QF"'@U&MOM=@ENMY$.Y=JG789E-=*J ZD.+;?R7_4G.:!,@D.1) M<.CX;W4[?EQ=D/[J_&P$]5'10_V!:%'_*9K]GAS3&)/ZD\.E/A-7CZW='-': M'_>81\&H_FN[NTC)M^3__+#Z1\%-8!! "-#S) HZYVRE@NX+N';=,#C6C[A7 M.@N"4_ZX9*'F4SH0]62^[;6PYF.XO@//2M=)QO.66%ET83N&+H91L!I0"P-? M9-V0$L[#K$>87I@>KN%B1D.[_T9:WMT>:KR7!>)FZ;BJB\:+(9+O0<_$$U#Z MZ>*@3\+('&$>H6F4!M&QU EW(_E,^&=O.FWQ9.SV8!']B=9N?D#MTE7 M&46-AEFU<>4UCE<=;.I>Y87.^,D:*V@W"IY(MOV=6GF10.OEIUDN8CI>CO5' MU!"MC3JF,^1>5:VKOT$NJ/.W(?A^MXV?DDZBQL[.(8?S'M#JOB8NO1_I'<>R M]]-WDKWY\*>:GHR1O1(OTO"_5Z6T=D\QO\^V"?#R]]D;__T+O^11P^^8-1O& MQ9>^1T+Z9_.M*WW?1Q?G]V^![D,)?ZNPPH_5Q_!?T0G\YL3BSH?_QI_PM_;? M?V& G'H&/@3NQ]]1S(J+!FJ4\D?WJ$T0OB_C]V#RU$B5\WZ]IY%2X9].A4<" M"*;U'<68#_#;93_3I\7W;&YKH1I)'I)_IX^SM^+Z]'MRT>_K^S#>60RTY\== MXIYUI8@( ?1A?B?-T^,$]$KT:'2+_,:19C9N)?#U=86,O\)W\A="/#S8^3:[?2(JTQ=N>'?E6NI2?P? MY5'<*+_C,G$=>QS>6@[::O,H#7 P@% MB5^>GCD;,!I4X)BA-PIG1O.F8L)7D&2H\+_71UL:@_)^<3^>921OM<9Z=*@I M03OHJE$1Z#X["MY?-4:A09&J'.AEHE(7HNYO1H%FQ"]OY590JW*P'0F&^ZNA M(P.-467?7=J2N+LZ5%%27<%M2 "B6:=345! ?-([ U#B:2T]P0G]6AY/]W][3T)*B%EJ?WLSNM.^%="WRQ7;4; M.7?A2^A'0%"X7<16J'J/;XQ>YW3V=')0KGIP6CAI,%\=@'!AIT/\.>QL4G > M@#X!2S9?2CQM46*,9^9TY4+H=+)DS$+G4@M>RS;\;&YFG3N\>A]"]6N9#7YG M&'4X08QG&W4@;+1ID&E@==IRA5?U+L9W"W-!.5A8?U<7?59'93Q8<]!J4V^S M0T(;:&LK"MUBD5?Y-!4XJ4OG2.11(7D4/IHUF% ]-'WBO>O!JQ&R7 :5+C5K? M7UU\""H-7!U[#WNP2\L[N8 "0R5T+5^?>]12(7M:5$-JMSJE2_Q0)DU3-DE2 M'V(_&SM_,GW[2,!*31PL24T<(C@U>Y]HA'%7;+YX@W1($$]^C6-$?6=]9E*Z M3G9_%G/830%>E0%[?U9[ 7E2?:YW6'8$2#0_C'^+7.U-CW\-2V%]M5]'>=%+ M>7Z:3'1??30X2KU\H4P-;3]P07!S-1)&4G@C1U,"%D0Y'T5M7T^"2;AP[D:W M6,U]NW"D1E]1'SP4.K=,NUC'8$=:?![74BE/5SKJ<-K3?^'Q$]E#IH@&\>HX W38=LZ'P1?,ERE0F90TAUO#U+=4D[ M.5X0-DH;I0$V?.9+$7E=?-@.8W.7;3U\\4L%@"!LC'M&? @VYDN30P=H43&J MX8U M?H%C@("!WDPL-A-[P5LE>\-RCX'S2,1'1QR*@#M ,7U+:SU_S3UB'!\%FAI\ M=#E]?G2$8T5_@E_)8F0U=0'J@&)2?GPA2S%!@7Q6?GAV0G?R@+1HPT[A/(=1 M_$Z7-YIFY47.3H-_'')B2[A2DGSY@'!R:F[V-8E2YT6!.>E%^DZ04FY C4R? M6.U([X%R?FQWVT4O+Q\R%:J?YY$S%+ 7/U3 M^FS$>6UE^W.O3[!0$TZ&6.ML!SM'7A>")$\(?Z]B)X%K=,$THX"W1U)J3EG/ M3&)A\3IZ:+]^P410.[P\D#115B):1SH=4Z-5Z6@;4[ Z5CL3@+\^>F1C5Q0$ M-#:56$)645]%W)+^'3I-M1PJ$PG:9I(F&D0 M/5]_=GM*@6Q?^E N@-)!4(%G?\Y@4X$/ U6!25E7@==:68$">A='*UH:;UZ! M&W]A@9ADW&W[+A=49H%)@)\_2X#^?SL;S6IP@10U34I*6_-/4&J+0D50K'B:;0P[;G8]59]N+7R)@8M 5TI12B12"SS#2)2 )D#? M-FAK U*_7IJ A&^-11QM[DF/:B5Y.'K1(3M"(WMZ17 [R$F+:D!O'(+20)@1 MU%Z;4,>"J8!A?&-M8WP/:&9QK%N9#2U+(E(L&U]\CE2*2CY\*DP47"M2E501 M8YQ']2X?!:&!K0C6@GU'@FBX5#ML\E"]5$%,:E]";$R!=WY+2F@C2%)=1ZV" M:WR80$J0'=2T'/,=@55'U7^0WA<<%Z#?]9V941A /^! ME5*P;8E_6WLNIHSG4C< =NR'4@@P>"CFW7>)%NF1O8;01I)TFI?^MN MHDE)3EU8,DXD:TA&%5IP3VUEQD:O4+Y&&H+X7!R"&$J%/^=1)$^]6EEAK%'/ M@ M>#%7);+II7$WR6%5&-1_S2>9I=EU%<"53'$PS@B=EI3SX=^1PB%,<5.Q, MACSD0HY@Z$BS.ZE6\3I @KA3)&1Y:!R"56>".11"TEV4@EQ&3D89.WY[#S5- M@FM98&I+9:AYDF%E#XYMR%-T62N FIO8TV!7H+N4U&!(DII?_ML:W]6@54\ M.8 J9UJ!*&GA1FV"+4YU>N9\$55Z3V>!DV=64VJ!NU5S9D^ SFIW8#9N'45( M/X""7DG@/3>YJ1GEB1()7-G^0 DIR)6)O=B$*7&OV43!\&'UP M=AI]0!LF?TE2T5U;3/5GM(*76[:"3W@R?^Y2KCK1@[N"^V,/"D&"Z$S^5BT\ MT6$:<7"#LE,T->!=NVMY1Y9PM&AB<6)\Q4/L4T-YRD,$M50X;#-]R W5?-/4D;>0'#2+T<_H+0=TI^.7M' M?VPT@0$%@Y) 6TRC-$YXT0+6;*%%4#V0-$,ZRCBT-#QZ.41\-1<#KC50-7@Y MCS6#-4!F/65* !$T,#Q' #UZ935?(Z [; YJ;GA5WCU5($2 FS5$-8\UVH+( M.^X]QSD_ CLXQD-I.9("QC@Q4BM)@#W_@M TX6<@-D4 GP.\,Y@_MT:"-/]3 MRT (-ALZT 'G OQ$0P ?*[,YOFAAA%@@%SX..38UR#N&-1%#!#9NA#%2IA%* M958D_SM.92@ H#D09HHU5!^)-62$&#IF-3Y&'S_X-2,V&CA#(+ U2@ ]>J@- M*#9-@5E/53P4-@ $)$=6+W@YQCG4/"P['C:H4W](T3RI0PLVE#FP 9Q/!D(D M?7:$8'FH.7J$14A[&B0;P41_A'LA.3]# /I5@X27 86$;D#81GL!40**A,M/ MC82* A):D(1/9 DY(#;>-2L!C3KX ?L%_S3Y*96@:1#KN M-KN$%C9835, 7(33/!HVLSG#A)$\%#N1 UD$ GE"-3X\B@(B3AYY!S4/#'MK MAC7T'NA6S(1I/,Z$L&9-7&PTT82I2KDYUH2T/QLF7V4I+U@ 6W95 #<_5@!% M /D>%#SN/=N$UB%-0K98J W@A/Y,"#;60($4 *2A'M:! HR'4Y7 MD#4, -------------------------------Cut-Here-------------------------------< ---------------------------------------------------------------------- / e||) Lyndon J Clarke Edinburgh Parallel Computing Centre e||) \ \ c||c Tel: 031 650 5021 Email: lyndon@epcc.edinburgh.ac.uk c||c / ---------------------------------------------------------------------- From owner-mpi-comm@CS.UTK.EDU Thu Feb 25 12:20:10 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA26029; Thu, 25 Feb 93 12:20:10 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA06397; Thu, 25 Feb 93 12:18:57 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Thu, 25 Feb 1993 12:18:56 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from daedalus.epcc.ed.ac.uk by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA06309; Thu, 25 Feb 93 12:17:22 -0500 From: lyndon@epcc.ed.ac.uk Date: Thu, 25 Feb 93 17:17:14 GMT Message-Id: <2156.9302251717@subnode.epcc.ed.ac.uk> To: mpi-comm@cs.utk.edu Subject: second part of document >-------------------------------Cut-Here-------------------------------< M(#?)K;\YTI"T."TE/@J'],.MM=75'0(M/04$ >KU:JTCI@ <-F6PZ#W6I4AC ML!+$=!=!7!UG.]+M]FK=<=/;'S? [7*SW,+VO[URC]SY]M3] 3ZC1XP74O7%+W?YV MU;UU]]M=-]9-+6OSW8?WUYULFQE^-\Q]@S?B+6]KVS0W"D #\-F6 M,*=]3HO3C70D+4T'W3GU;DAI>]-?*1Q= WC3201_4%5WVB@ QIQL]$QHU>P M'! :MH) K56SVWLW'.UUI]Y9]]7-=9?>8/?TC7H'WM=WQWUU<]^ -_;==R?> M"CV]1U^:]^F]]\=ZTW?P!\RX3;-?NMM;G?0W?1?2TD MUE- 'G"'H $;-FH#%F )Z7:HYG]SW-1W"0Y^Q]S5M@(N>B/@*S@ OH"7!@WX M#?" (Q$E@03^>H_;[S?E#7>'X)0M]/UY2]\F>,OM@A/@+3@1'H1[WZIW]VUX MO^#K=I;]W.%F;3D0#7B&0-V#:]RR M]D=!:Y_<2C@+OH07X8:W"KZ&O^'_-@.>3PLK,SAI,%Y(X#: ^ST% I3!N;- M1A\!PPE-C6>[W97WG^IT/]]D^!Q-2Q_@0S@ [^=A=TN_>G-H+SW4HX)&Z$+]N,^"H^@!_ACKC M3;)MVPMW@HI^Q[O<@@2. [C? ET?7G3W!9H""^B'6[W7PD"H1B_4/K@:#7H_ MXJ^X&JZ*B])7MP%^>C/CK;B\/8NWWF[B#(XVBB@2> [@?IO5C8&6&HQ;#$IU M,DZ"I^+,MAONC#?CU/@TKGT+X-4X.AZ/4^(Y 39^?K]">,R9D(MOX@[XSK95 M$R@;=AP-*)P%I39EW15!Q>CW/;WCAU/1<(*_EXG%T2. JFQ9--%CQJII'8465OTA*<,- M/X:ZUWQOTG/!^IU/:\BC=?^- E@%Q5XH?4L7A)Q@7#U*Y\RW-!T3:5O>FS52 M($R_TGGW0TW1\=]6=4, 6YOEC[5@*$Q/W(]UL*A'NP "=:C-T_'?M#18+@-> M!$1W' U[NXFA05ZM=TK+ON%<;7GT"^GV8PV7V]Z3M# =\3D&,=T<0%_?TA*B M I"0/PQ_M44 6ZL@70!>WK:IU8!Y0IX"C '#M!NP>3??M[2)'9=3$)NU"Q*8 M!]/+MFQ1,<3590 *8):SY"0!N"V7H]>WMS"]!)0!7<8M77$7V,'!.]=EC-(& M0F ^QW6_JT%LWF.[#5L=F/=8OZLN]2A]&7SF,T!H#GOS:YJX<,V:U]*;-6(0 M:6L*7CDI#9O?TE/&K#AO4X#"- 1D%,&+0$;^)I;Y\SR;(Y5C])F>6Z>$@K3 M6'DZ)YJGY9AL8BU"= E+B#J=F\NKN_C]&%.&L+>GT$-ODT+BK"UYP#FW=)WP$I@!E[>;Z5K.,H6_G M3?I_7I?GT3WZ8_VCN]5_:IE^.4IP?_0HG: JY?_X>7[\>>A\-'^ -HKJN?FG MS7:+VE0&J7UFG]J485Q]2[/:4L:#K4T%.@\.\4U2C]()0K2>4B_;U'JJ#5MC MZ^KT+6UC6^%+MG:N:JOG*SHI+2MNUO#Y*.V9W]*(@=)M H)Y8+5],QH@3%.#/X(7'IQGUQ%/MXX"1 C.>;6ND*_9*$"7X%+[Z=EY?IZ0 M.P:+WV9-]1$:01U>'DB3U*@YQ%YT5^MP^ACP"@;D"OE7/F]CX:/TQ5)2G]_Z=M!E/-8^NP7N$=CIJC9J4ZHB"5-OM1C>&#K6GZ(3X M#MX@%.E1@<&.L:?MFX::'KF;[K=ZI+ZBHS:4^T6P!N#EG7>:#N8Y<&QT\6ZY MA^6?A;4QH\/FHW3FQ[+SV7C,4854%^ZDN)<.1U\&F[+.AZ'_Z7])-0MX;EN<:N(1Z8+/M?KI6SBQ7!E.#V>>K2Y6:&;,> M;@_K7WJYO5V%0K39ODM;;,W MUU@[P;ZU?^47NP6/P3?7)#R8EYL[[EWXLHV%N^H*0N5^O.OPL@A2?06D &L& MF!=(L^R<.8;NF?OGU#O-C'#0U(1[UQZW/.SS.HX" M4 E]N)_NE:\H8/47;\4?\/)W1!VT0])X^6<#1S?G)[NYGEV/!D@U'\_(9_"7 M=WY%2[_ICKE]T#\@UH$\F#=*-P8H 7[^F/_G^'HZ;;8[Z;ZA=JZ_,^J;NYJN MFK:8GY <@[%V9]$%_=+,^@3_K M$_=H;O"A\1:W0@Z5YS8-?""M=@?:O=SJKEZ'XCIB[(YT^P5U-]/]^RGS(&H- MGD\OC8_!XOY8N_#F^F-ML\/P.3.>'9R3WHH\[(ZW#]/Z^YN.F<_F%8-O2Y"# MU6!YSR[% ^CD_-)]=S\K0CM;SD93Y07Y59[0V]V N;O^MD/EC[S"\9[[! MQ^^XO"!.2LOHI7S=+L7_[R"JV>U)A]OM=S>O'13B\W7%]!+[W[ZS'['IX%(?40- M5N?KY?S=711DV0B[P3ZAP^KU^DY_1]/P\@--#4F_\?"Z,J_9<-K1H0[>O9O8 M0O5V#E8?Z)!A,)V;8]#I^;V^I]\3FW4N_UB+?U9B\\V?6^D]_?M>NP_V;H!H M'FI#]+?(]'1-P@?/<8>RP_3,#Q8@+_/ MI\M!+.^UP^A]_59^?&#Q>'S$GFKCY1XY3M"[!>7Q DGN9 -U9 &W4&8T;@ @ M==V2Y]/-]#/M94_3U71-/F;?Y&;V-XUF[^04^^M-7;_9^560;CV6D%@7]G[!?2U0UZG6_2.>5+0SM_2V6NJC5#'YSR]1-^>>P<8>ESO MN0/=+0 6'HM<;YHX%W^AO^_;;3]]GQOJ@CU9OI4?ZM4(@*^V#^VT=,4MHQ?7 MK'PF$RS[$A_AU_+XN<4NOE/QON$_;^!7 MZ 0^)(V%1P:I^7>>IN75?CE:;[LO"O][A["/PX$).UB-$$CY>[9O'[5S"45\ M=2ZAHS:\<]6@OR_F=+I4 *@+\8\U2A_GZ^I5184MQ-_26W%ZWL3W]2RVG<_9 MD^AO=E[-V(OLROMI+J]CU;GY/7B'6/9V(1%?X,?18WW7GIH'^:[YI*^>8_G; M@9\H3.?R87EMDZ+3+Z#[1 TX#W'EO>HYE)?DS'U&T+K5:N?B2EZH01DNN98] M!,3DUSTTG=V+V60V3N[=Z^1($\7^>X_W0+EY/Y33V4;YG5U\J]1"=32OEW?H MTCL?;=KDXFE!;FY\/^O)]R@"@C??TGU<5>R+'>@]2% HE :5R8/SRKDB1]QH MG!8X^.HT$@%'_PB+GMGQ;1?\Q#,]#14F':EY5&!C2QE\.1NM C %9\ 8L&+S M>9+#7ZT"C/PF)PEQO20C5$+]_+@!:#_N[+9_$U M]G:Q4ROLD?G[_M7?TKL!8O#!/_HG=G NTX?U3?MP#WN/!@X[4PU T.N>^9E_-^D-GIHPV]_IDC\<3_F/#T8 :L[QOWGE0I8MQ/?_^C_[CAW K =)H^6Y MZIQ_V;\925[-EY>;:R04=U!_4X'(7D4O5A>O@U!4#>QT$CY%'?$/@[:YD]'1 M60=33@WGZH@P<9&>\Q5 M_U -, ^W7SJ9#8!_"P(\])\=KU;FJIO7J6D2PN@YE9GF3I(7E:M=V/:*,P= M_&)X9X !(!*P]49A"\Q% 2-ZW+^[VSOO3D>02_7-YW1 6@$0(!BPZ#>GL-TX M:D04V#5.CJ2FZ;<#G."4.:1^QK6QGA<.6157>Z=I4T@V%C6BW":G5#.W.>]9 MPN)=9ZLSH!F-DE;T^PQ@ B-^J(*)GPJP"! $8 )( 4B!1 "_T!"G]99_6_4= MV@QK0 &6'4\@;8=!(\@E./ 8V(V_G2:PY]=FRVQT<78V%[6G0\HK"" %. )< MO9 CK;@P5B_K)R<;5;8*!#?_-'<\AM GN!GL!I8"0PFC,*O ;2 M5)* ; ;G7W'-+]?5,]EYX%!V0< S@2[0"& -Y 5V ADUZD 00#"0XO<$6 +0 M_OIO^;0G0(0HW;9>8Z,5?9Y_ZIYJG8^H%]@'Y--D4\0.-[=^7WZEYD,(1.\- M^XY[(KECWW*O!* HRPBHF 18\0)GWY4MVE?=Z[+-Y*Y]-KDRF[I-N@;>HZ7E MXZ&0*>TY9Z=C>K'^=&PI9E,_NY M][: ,3W:'KU.5E?22]OI[&Y_2KS-7X8/%+2RX_OQZ_A[MS1%'NHN9Y:;P\*! MU?1Y< #DW6K-:)>P.[/A!C=KD@-873WOL6>&N-"9Z7)ND[L(VL$NI"=,\^=5 M[)88K+KY7TVOI1=I>_NQV"!HWJ]671;PUI>> PWB^?!N<#3>H,%.FB<;Y.W9 M_GZ#R[;@X$5OF/1MRZMA@58#Y8<[FJHND::Z\_!!^)Z#BXCH(-?NCO:;0VW( MJ\ "6$#LH 3-OS:@@Z-!]%!SWL%:7T5 IE.9,_5YYB!IK$''P?6.CZ9?T,3) MY7YW8#];G^UOA^8;BN+]['*#8,$D@)7.H]9 0+55A7Q$(#:P'&H#O!4!'*5) M]KQ\Z$'1H)H/UJ<>X!5U^)B#8;FJ@OR/;+=+*WM(9R_!= M"*& ![TE763//H"0$\RI_X)S([WC ^BN3H-\&Q$:UBH#)4*!H)H@1;C?\XG- MYUA" ;7'6F)@/O=5BP[9_9IT;SIRFE BK\:R&^LU?V!O,@$3G8CP=-?/>]>I M(\8_P 4M1)&..;AL2PS2\42#Z#\!GPW/KE?68__-]*I"'S\>GQ=0!A05I.,M MTV)X;L"LG.PN3F?HP]HM?BYOV;7P1I8.-O>J8:/-A4QY*PK4W#70["#)XP^4 M<-HV$K\4( K@"I $8 +\O48$63;4G8=OLF?\Z^GIZO( Y*CQ'VKC^( :& ND MYTZ#?D(-X'M0,:A-B^19Y3!9G;NI'J/0JX:]0YKLXX:%'CR/VA/0E-?J$Q3N MA[2 =\#-VIA@/N>KTXIX O-J$+WK'+O/8Q<[^.#AYVJ!\[]06@,01AW;@.M3$K5-NQ M[0AWD"(9G]ONL<:R6T),#-YU+"' 7+].1N>0\_I9'J1_",-O8;V0"_@"#%% M]49V]AUH8)L-PJ:T:PB.^( VF @?H"4P"(@9@8"= ?^"@0Y+VH1M"C $B.A$ M],8(!#Z2X6S.'!A'L]AYUF!K,<(=VAA0JJ2TJP8N"Q6#VC5P(*$-_X!6"\71 M,

8#F6FS0=*>=ZQ#JYKQ_B\)182&)O,>RN=N< %\YXP45(*O05;@8M-U M1<0."PY+X *OEI,2I SR:31JYS1)@2)P$)>OLP&F]P1$@<"Z&.<&(9CT 0+V M;E@G!C\C !MM4Q>TVZCU &!KE[C\"BHMRY86@@_Z! QI,#@4 (LP7M>?2PZN M!GQ#M\A!0,P/O8P>&!;&!C!K:(1X#*/C*$0:2 HN!QT!\H -O&3B.0+"- M Y^!FK^M84S 3:0 \!K*#7V!-@%$T3QP%$@$J ($W82'D#2FFBR0#-!E<$6P MT6QV=4#NH$^/[Q>["P(2541U[4!.H%2)47,^G!ZFO'Y^0;\4 #(PZX>K2ZF= M_=2#%#[GWRTM(=#.F]!-VNQ]LKH9R3*O=I@^Q!W6Q9B'[L,B0!- @?CXFA]F MV<8/S,+B&OZPU;<]!-3Q#[N$A**!G&_H66'TDQSB)P $@L-237GC;C,9+ 1^ MY"""QC[EWK,BV<=DT+*@)S2"T[TL6T=0)G=[H\EA^[A[(\'O7K>/EB:#0PGN M^T*(4"$4(K\O,@@3!"S,!*])202;(,9/X<=&8_BUV9PJ:YN?X!"1&X@DC-VI M'11%!"%X?D%P05-03(>A"Q;"!;$=#CO"8%:0D496,P+0 M_&Q^.#^=GU5N$S@6+ LB"7]V@;FFX/?O+9@" S&!<%M:D3#(+^&EH8T)/C! M!0")++_"8-3+A-_JU M!Z=R^L(*7H)POP?*\]J)[J67HR0K^>VB]EU&8Z#[RO7(4G'U!<5. ' \- ,'SVPVNJP\N<<7!B^ M"3=IHP%N7'C-:PYV M"#UP<$,0H-R0)&"[B2=.<*:()0&]8:OP[_5(+"@R>X2&H6PFR@30H[8V3:';38C@.<0=,A_&QU" MTDJ'2+$C8.IPF:BYR\UQ"#%T[T0R(+!N%Z@^A!X&#H-OO$-1X# 0>"@_Q![^ MO(B'S4 XVA,P"A@-) /.!)J'/$4#XH(#[99 K!Y>#\&!VL/9W"P0=0A' Q_Z MAN!X6+ZLG0?N A<$# R8 6V'S\-LH%E1'BA4+ + #]\,#D1(VM@06-.CNQ\" M#F!KO+S]H2=C5O<_W/HEYDR :#?GH3MP?3C!>2HF$!>(38 &HE$1@@@V-*Q- M$&.!6D7NH3#M@@BGRR#.[SB(!4&V29L&Z0-$S&P!(< GXK :0 3P#K(1- Y<5A"P(< MO3AL10"S8& N+Z*QP6 M GDH%T5RHGU/@,4DRN]1$#IY[KW3(#PO]]<_$ '!ZFQZ@SR*@&NO4/CU^_"! M_=R)!SW9WJOO".3:V]'MV3"$S<']'I]N?*BUV_5E"_EHG)M-(9C'(6?B2R7V M$W=Y83SC7>^O3N?;FR0T]O2%XD)[86?/D$:@TQ%@CYSH71N&9@NO*.1 M[UIX[L(&(+R0[<='4W"8Z'Z#4L,.'K1@-C?;X^NI!P>&[;Q1FE=Q0$=+>P+H M[1)K?KD)8WIN@KBR\6P., M[ZIXY+N;GI9Q-G=F5"76ZA1-'L8.W]<(MM8FS"A2\6!K#@)0^">Y4P_FYBB+T,0&XY"0Q(A 6]R%Y1P("+J#@3JMT_?54R7* M\Z9V L,WX]UNT3-*&^49[ )YE+\]HF*PUKAHU-4%!#L&>+F?H69F,&BEZ_RA M!E%V^3L&X9[-B'?8&QVX&3]V[@3'W^+.B\,8CGDTQQVBY4Q)F_T(U>;7_W-F.-Z#KFZ_9Z:)VE<:BFP;0.=AH%.@- M?PQIH;V?EY50J;A94^39![.-L#I2070NE <;X$V<^0)[HSC+6TW/ PA0A!,V MA\!M>:!P8UU."HCE\S8.'<@ ]+5^'6S.(1?F0T4TUQ)\"8&"X5?@PC9GA*V% M"\H Y[EGXK9NB",#I"9&=3B*6I'77S810* #%!!9#F5".#5PHAB136=/A*0I M& %S9[]SXFE0G3C;Z^K]!AN*)46[P,R1=1*2TP$J!/>A%0>+/L6Z8W.Q=TCQ^QT: XN*M#BY(E+1>.@,C!0F M#X. K0B[8P'Q'5CO&B\D$-^*146Q8?UP>>=8O"OF#]5^W,85( #1%AA\X]P0 M$&^'C,= !]:QK<@$> (, 4J!*8!D3NNM0%=!=K/#IN*LH,@0X!8B6P$!A\A ?V%DV(HL5]'VD1V4<1I.04 MD@I);I6LP6H1R_:2V[*]%FN((,'MGD@P)S>!VR&RT3IR/[F4X&Y1B.A;W-/D M;:HR*$#?34W0N)CP0RXZ$1U^4<3>35"1BO@R!"[8V&X1ZT0.VU30--)=#"3& M_,*+;T2OH!PQ@@<%* )@ : 3P I@- /!9 7[",B'=U$%IRD7R&0,&=@[/>I M.V'H;@TXZ'P7?=E5/#%^-R&48%WW;Q0^=:HT^2Q M;!!XASNX'&K#"F=,_-!) >-_-#:Y7_I/?!ADI!F"$4M],,:C3P^/-0$S]*BE M\CAY/CH-G#+(.8=[J/ZL$FF%8 T"GZ%0N+?$:]+9&FV0*$.(@09P_0=[%.:! M[IY^WR-P6TQ-,K?9<_&)!J^#LJ*HP'YH8)C+,\VY['J,"#9[G9&PI5=_?.6- M$H&#ED=$WU[1A,<3V-M1"05':0 =1(5QF&BFV].X!V]\JS4U8VQ.%LCJ\^>A M-B)$9*$U)![O*S!+] #R(+UZDX3_G68FFX=1]&UI%!N'CD"]4B0PNR8$!!Q6 M#B^!-X%,X-VQI^@)3$7N':%""@ 58#WP'HA5/-(5U]2,"KE;&NY1F,:\V,#- MYYJ0G,=U36 ?0AZ)"SR!X:1CT>@'US1J#A7U*-)UFAI$\2\8G6N?ZAY]"LJ M#^$"Z\#%HS$2&ZD 2""6 D^!08!48.BN]38$^.I)$"V/%,3(H@5QKTCWZT,R MV3J(90)3C9RBLZCP RWVV8A]R+WEHVG1^2B_N6^@D:)[5K888O5QVG=]_+)E M'[-]W3W:8O>QO$%6<\#E%L5]"D'UGEI@$T? D[C!WS9Y_,"\ !NM_I882-NU M KF C4 K&X<@3G-\O"2&-P2,PL6,FMD "5)&HO8CSXK 1 =:+>4$HX'R1,I!'/$JJ0L!M_-'NEBBOI=!JE^"\8S =9/AIBCZ!XZ MU 9WLD9QGHQPU,B98]GYAO9Z,CK4'$<@9T;0J_QA&25ZFT0YX-L0#PD9F H@ M[QQJ^CW:'[_QEJ:(%#-F,:QUNT+8FRK$8?<;G-!1'.F, +T:X4"/OY>;6R:" MU7)Y]C\IH%AQUHB\FZO-"2MX(D@J9'IN]@C#X^6]&F>#SS_H() .IYC$N\#M M#^T#?[\W35ZMPBBV2ZP)"3U] SG<'6(MUA@LE/4)AEAV2(&/WX.O8J&[$*F MYQA[M3Q'I"LQHA:SVT%<[W9W9;GK)&RM,PEP9"5F,0ARF3WB'=R1Q3AK3!3J M$@]ZF4+"85[-.0B_D]_MAWB-I+S)'.51%OF0=C\E)5!.* &^Z2PJ M..Y;)4F19/(Q'RD17"$V'Z%L,A^9CU8$,3%]Y BZ%CV"V,?8HO9QMD@25-8U M) -M#TFAG$*0M^AT)")*$B&#\4@D8DL2X8<3/.+H!'F"5QSFHN_&N:@"S$E^ M!FB"6D2?I IPAX9=Y*E9Y9B"'[^BX"N/5+=9&_FQ*_N(V4HT8O/0NZA(7$IR M!>&(7\%.85C0"1 $6"#*%]&",KKZHAY1F+A_O/4@$K^+6D&'6B$Q,'A(S*N% M)<>2=\&L)%J2D7843%+^%YU[CD&>#61PT4:M'"(ZZ"R#@#0L UY2,_A)=."% MT@I[B#\)WW[OJ# 0E$/"UG8#M,9B6<#P2$EK1&VP]9AZYSSV7(:O-"E,A-7% M_A!ZO#/?V1.(35?CTQ<*U>Z0P,2EW7R.(] X-#:N!BP\3I@]P) M$]]R]$GB'VYP:)=YXZE]Z=!V9;FU7=K.EP>%G K,_^"%1LHZY))0R-@M; Y) M$U.6,T"=H34Q5)=-]"R:(DF*_(&CC\XQ=I^NAYK2&_3](6KUJX2A,T]DA M^N) @;DVGF9O.L )XB<>";6&8[Y8X\0-*WDBBXJ)R\!C16RKJ M#*6!#HZS8OJ249/ 9"M2$5U^4H"KXNJ14VA8NRFB+F-W33K>Y&;--QE'PP/@ MHNB.#K>H8NC1LP&^?&!"'B,AGDM&)0@2CK:JQ#Q2(RF5UDBBX5,Q4SG"I"IV M*C^5D44@-I?ZN53E9W$.*+6.5O\=ZU_CR%+G ^RQ. M!B.*]T@AHC[25WE:Q-OX)--J[0.6'&M1AGBLI"$:))65",D<8FWQRT%+6R&\ M**N7TLJ5(/EQ AFO(-RD)+.5Q$4E8OO1*O=^%&?%'U^15$0SXA>1>==?A ,T M+/N/^D7P(E-RO!BPG"/V_!"+W!(I *;.^G?0TS_B*U<(IA'\(K]RO_BN9$LF M,0.,;DPV)J[2Z?@S1# B A6,&\C$HZ@.DF;VD=#U*(>V3\S;>@O1N1R>I[.MY$J^,![W<'!LOWAA' MZ[8Z5[M081\SI2@&=+?=)MU[MK]:8)E0@WEYA*WM(AN3ND9'89Q/XZC%XUEB MZ !S.[IBH]X20CE,$C52,X68PK1(X\2 OD9ZHQ02,R> ;SIA9FHR13E',Q"& M$C^#XT(^I.IN[>?F.3[(^,*6Y+O5'L:/< D(3CPYU@VA*.-+M.)N3W391J-Q9;/.^@A'85O6D2BW%?*^%BM)$4& M?;*7BD K9?<22PFBPB82Y2"!(\7RI?PQYA.F'%.J%,>.+$71(3B0L9A/#,5! M"[U_OC\OGS-QMA>GI(EQ!A>/4!M&C9O(;S-_S%,*,)F1!,SCH>%14!E\PVP0 M(].*8LV83P+QO"C!!"52'N%H%DQO9@;S#9"^XV"J,T-I9<4F#@W3&'E4,6J: M,)61@,=DX.0QHM?"Y!&^,/N*X(8 H@E06M&_7&!.<.(5"41/Y0$RASEAVV&V M&GN8'-)&+V)]MYL$HPSTT3J4F]Z2R2;-)[O44H)L\&'QD15"&: MY"HYD(22!22A'I?%?/91'Z5]U#[8HG8OC,E]3+.1,<%I/CD'7;@OC=FW6V,2 M&,V/$LCAXOK1)3G'_%8J%^^8X\KV91]S#G!__&0U( &#_,(!.,*\M.8NN&=9E/HU)* M[\Y^QTM7)A,/L\>S?-?)YA)RTP&V'4%/$>3:R\VA[FQ_Z[]=X4N/LR>:$ZI) M]=Q[CTGG(&* 8^8;>OH)(1UU7@'7GG9.G/FU<_.\!KIR@4M5G@:0K/"?Y%#^ M?IY\]LPWW9GME"FV3&9Z# >7YT)_IMN0\5?(0R=B(AMU0D*YG"<13:>XW-"- M$C61W U172#-O_F)[%Y:V68"G$J)XMHF<$-27)48#36'8LJ48ID2!6!V="FB M .27^D3ZY700/W?5+,L5&^6.6KMWW;10F%9OM!'.2$ #<\IB)/00SGFG)%>^ M#Q&;B\K%YCGR4>G"[&7",".;P4@V29P3KMD6N-UD-G&8?,I1Y<+OL\G74UPF M^!J;MS339J227(C$A$=^I72 WXW.XE>J9 &EC&*.%GN5M\V#8FXC>S?Q*U:V M%JV/R,HOYG 3AUC -^Z4=TY;9RC^E^E$G" M'Y&(>$QA8&23!0BKD2[F-!<%$"6 M%S>!Q8@I !5 O1F8LQX&YJ8 0S\')'SSUHE^?&X^*6.;!,:O9.00JX*2S'7. M<*!]DTQ!YK_2VOF4W 2^_+:=4E+$&9^LV84@WN4>@9A*/=#^YH5K[]I(?.HO>=E/$) M"94_YDG\G-=OV.EE\,#E+0^<.CY'W_B.P1FOPPVJ(%6,<3SNG>6M6RBM,-%- M!A!K+4/JW:+0")" ;!($ 7!>;ST(H4NM*"%0) #B]E8"#C8['0WK?F?C!'DB M'#A\;+];&LDS>0GB[-B-TER>'S]>8722:@G/"Y-[<+1[JHY^^R7]C& ]M-)XE_4\^V8WEN[?GB)#(V MX42%637/9?L/.O5>^!M&OUU.\^>IU@2Y_4!A*1Y@8:)MT?> G_,OU:VE-11 M /]J:0*L6L<3/4E1RZ;L*T^9<<.>W[U3O>GM#'K:];J- +V5 '_LB,?+;!18 MZ!X^";[J9$53[QEGO DF!F9LBT^Z)TVM6*C#8U*"VPIT@30C7_"R5>>U4W-: M%J^%SD_?UO&!MP;:/!+>(=%\/KKNI-M0&NG+$[GI+&=S,CZ97:Q1YIEN(T5> M\TTB6^J1Y!:&78VX.EIEBG#) /TMZC,^Z)Z30-&?">S_H M!UEL\L\X6E]2,C?WVPZ./#^8MS27IPYO3H.F.W(V"]]]UT]]H180R5CC#%S> M]@B7/TZ"W%<.<4G\4URRV(Z0A[L2Y*OR82?VPS:H'"?LK"=B%;UIM+2.(\&!=/G^@0#BY^)Z(#83FSPS(3?6HZ]! MTH:9YSPCWKKPT!?^/(#&\ H'ACKBY;RP2,G?8U[6N_YV@[P/Y@!4G(GR9 TH M.'>%+,\[&A=2 XIXC ]Y#0V>*DW%)@L"=?C)2PPM/X-%T$^"WN4S>AE\+ GX M0)V;'Z7 S>T&J#EU?+^!(HERD1! H$@RT*$#)%_>'!-$MS6;(*$3$QKLI >^ M+V5^$S9;9.6Q,IF^VT7J%<.,C\@@!8+R)FB)/!**%GQTQ3A1U?QO$&H^O'^* M,,.:-H$KCC>2"5!Z' *D'HV*#B:_)1SM,><6M)5M#2L3D[/&$R(_60?'5F M-LZ8RTV58'.S AF0;&/")=V=<4SV(T^MURD"A"(".[.;0L6<)+_&V"D/Y"*J M #%"RTX^YG81VNG2DG;V*_F+XD7SYK53]QF5' @ O3AL6@#VHGOS2-F5I(DZ M+ >>-]%*IA7Q8UG% BX.&#NBFTR39>O&%?')S&]B"L<+HDSBF]E/=$D";7/^ M&.-U;5!UVBEO":BYZ_RA.6F3/\CD0&(-<.F8$UPJ&0N7*E"[$ OT-,@!M%EN MZ%R&J,I0VB\3XLCNZW_"U%1!7$4VVHKQ.@>8NV!>"PYYLTQLXYKSZCF? _]\ MUO!U];L9$ :MRS#!TTP*1D6@&D\FG0?.+7K0U,U1+5MV_+WCH(EN#E!I.BTT M>%Z6P]"\Y9$P+/K*FTT*)C%"^,+_IST1-=>+3-CY/L^?W$G*0 [2T-?Q[#(6 M\3B&T0,[9&%TG#EWM%GN&<%M"T).T-ON?9=_4(Z.1<<$-$^.YC31HVEE2W!< MUW2:-1\:I5*S+0"BPK(105.:TIK6V\"'%!K1:QTR)G%[X :W(6#2]VCIQ"]T M([&C/$VIJ,SGIYDXU%XV.4.2#+LF#DBQ=J._J55N1YF4#L^P(YD2JFFFE&J2 M(]T*A,:1*"T-(X3Q)'!Z_;J:8\VW)MY1?3D(W"9J-_V.T=!DX#044+G6K &: M+XMY*PK+9A!'Z=A6](9*'L=T=TTVFCB4LCBIA&QV?BR5N0W!I@B3L*D?O6&Z M0_MOHDIX*)"ST.?H'&VB,R.="H(B)FKSB*G:9(IF*=^2"4&B'+\&MMG3]'0J M'T&=+$0'3J #%QCY:H@.)(.;R!&6=ND5:)R*4-^<4?6.J M'Y.((E'4(1W3UVG'/(F6-2E^)$^UPYN-N1@,3'96)[&+@3F>FG91C)AQ_"(^ MUD9^=U*C*""3X-E&;$H6,@F0V4XJ &(Q,,+ (H M2AF(@3G#X-<33\IAL_MQV/ <,%LBDV4DOF]4RE@JY".7PY#Z,_PRS$F+2UM M,3.0"KT-I+2",TC*#"5>VC*CL+5AYYA3F*;\W,HA2!&<*L\%9X@S8]@EA/NE MT99MT,ON:+YP>1=JHR=2^ 2<8SYRVO"MAL>^[I1TN43\7(;S76?V3,AA M 6>#O#U J(VQN]!0HZ7Y19=J.40L**%Q7%?^U!=>_L"@)LI1HYNPPZDX\7'X MYF1T>[TXGC0SQ8@Q9-P5/?%_ID_Z19]4B ?'L\XI,WVA&%/4H/MO_TDK/'QV M[@)YSD%^Z5KO\2F04WD&EORE>3E[YO).>(B:XWDJ($>?.KRO5 =29$KX9'T^ M(YN YL&67N_N3PC>ZI/&/M6%/TY16NX3"I H163"_*Y>D8$LFV>N^7>[N]7Y MXJJ&%,=>9MS.)S"WPVK^!I-UZ7Q5/,#2S[-P)#)>F]TZG:=3K-CHN7!DR M&EV#DR $'[L4VV@N3+5']N0!,UA\A0&FIN[??^3,C% M165V/;UVYH33#)G_C Y,/UV 8$'6HWLO %H#7(/F)]-SVL^OIP)TS=D 9:L= MTV9[H%,):$=R [I)$_4Q.@M]%U"L&HNM6IH9K1GBU,"%^L+7) F4+7KC?(NF M0'5[5D#K6A3M1.J/OR'NBY9*K]*=EH==.K)@F2JR DW>]M M&71ILST5:L547A@:\H+R2P6(W W%XS(49I-&Y8;V'4N!O=!?J. M@1IK L05PN>1T#E-N8XB.T>!T%#.YO#03UG #%1Z2,^ABB(:)BY4Y>)'[8;B M.;^A*%+&)CJ2_9EY_'.^2*4; L0YS1YU0NH3[([.-2.8E32C(NO1L+;59#&. M%8]N5SF^ICWTEO;7/(:BQB2DADHD8B,UV5D$T&R"*HV*BLXFXN_TCM8C%8?B M0XV8=;\B:0]_)Q;ALK*7"S(!D1U9(R M*W6(#,DR9@\13 J1E 26^^1]SSRA6THSU):!\T5^X)AOCKR?(8@J/FHX_ R< M -U$I,5\9:A&CCD279,V_"*9GII0#4J4BJ@2513] =VH+U$E)!H@)KHG-;J! MV#!^"3[] WW-P:E. ZM]"H^2<3:D**]TWNF4-&0.!+( -3\DP,&262:CN^6M M!6=T8 00VT.5)LJO4?EA2I6B%[AL)*J],Y.%60>$+>Q/3/$CF76 X.25>?/(3S) DOU^C\]!=@ M[V8":+HG'H%/FL=C+%!^Y4 $EI^/H=3..W!O_-GP_")SHTXVFG4Q?6?[4P[0 M$\R<>\\-W\W.C>?ZK+M-&1@1&#K8)'^/,4H^O,"I"=%T2U4"GT-N=BE!DS;& M&E^A,3OD)&=R:DE,^S'J)W&CQTR*@-F4^.=7]3-2/9V>,Z#!:JWQP;A*S/Z% MZD1] ,)C70+U,4=9/4.8ZPAW,CSOUPO0NM9)2]-A^6B<6;WK%KVN;Q2'!%'& M5G=ONCKA),LF+FIBXWJ2WO*:,4+:*:.0#D#J0Q:">5!SPS79JK9N78.#H?-= M[P)II=3](:!.40@"U#]83(6>_T00X*?PO+H A0)" Y%WH;;,Y)EM-7IH8:,) ;J'M;^6GE<@.@=6C*LI3X6,;U,J:(UP;)=0/>;A M\LQ_P\05'_0N/W@[#8/V!,9S@,M=7H8/=X 9H*FU[KJ+507B (R3LA:ZC*7A M 2145J*#JC@S-UEB7=0%1V%X%L X9&B2L=IB=$^^ZQP$=R&V7=XN7G9>LNF9-NC./A!OND DK3W([N71<&7. M$EMZ40BU!GYN]Z@/5=U%&D-I8+6$0!O22;A?'4%*Y_*E?<7;78(2[:?_B4B] M +,I=L<6I>0R?/FSB1RN9$Z 1Y7;&OE4(OD=19_V'(&J0$MS(C@3/;I.M.>= M+KV9;I7CWL_&:VA>?5UV!D.)I$I*H7) !X'Q/$_Z^=R47K_3 :2HN49ZTS_\ M!01JR9RZRR95?4@:@ <*B/:IPL"7'RU2#/ H/..M&8*CP[@7X7GPH.IS7<#DAAFMQH\S=P=(%]W,0UW7H^'&RV6VT"'%<$ M)XFLRU66J\=TUE??BZB=*%N#]M(40/:SRY#SM,XM"LVK(5,#IH#NX,D8_+6* M'8*M)D3N!D]5/JH0.99V(J-VW,M&H#J50Q#E+-5@>Q""AYJ;8Q+5?"H@?6J2 M';F<9TKQ*)N2?7I./%XN* VCBQX!(N9FT,E)G=MH-NJ<%]) :IZS=+EIE+NR M-1%)51EE*!\5"<*I?&!^*GN>U,[6&R4/]S=:X=!&. M"P6(1QS.:RPUG1<@2%0N(T^D8K]&)0MSD@JI9)%6(P&=&T2F*(@JZ$,3-!Q. M7Z.BUM==96VSM(@0!5;>;KX;@1L^#4!2B_G;G"%6^VR(LL6$9+,RFPI.PRUR M4YF;Z;UOZB;N]0;-*T.FVT*1;(YB*13J+;E6/"+6;+:8W$HUJ77S66$2_;]* M42MVHX.8J%5N,7E0734RR^0)#5C]G5OPXX=__">>)8$(.\:&:H;Q@D=2K=.T M5,F;%-5#*7IS"E!+)0+<'], @;DP@'[U!60TA<"6YPH('[\$PE%2DGE2A:,M M)L&;F!%-G%*29"A/"'C&;+;2O/&P[%>R+0L(>L'](]H(L+>#W8$.FD=.6GN2NE%H:K/P= M":M&&D!,7= .-?<"8F;^URRF7\ >; LPQ_:_NPETZY29"02E:9;M8M$@H.== M[_IUUXE(9/@P#6N!6\.&)V-S%"4USR%Y25(AWW-,(9 ^*J<*$XJIPH6CT/!+B M8;N9;[KD)YSU(B5M;'Y:[]9TF%>^995P&$=HU0/V8$.F D+I*:!P:8H-K*H= M9"]J"MF](4,1%TC>FIV):5I MOU"&JM#A'RPQUBBB;+2>\T9_>M:*G4#U%_DBG;+BT=BGYD\;J ;P](DQ?,6* M!FN/&U %X*AQC0,4>D8L>IIT"0&EP%ZQ8RJ[@_'E+U6,9A_ ZI+N;GKB&S4Z M8PN?E[[=7J0NGHF M3:(/.-XKP)UK%Y4$(C:J[3!T< -W-:E:Y!4<[=L8]G- MZ32(O,A\!O+N*$C]::Z1[DBRV()-)%0HM$-=3,DR9+$%<<+)[+$3)BN3)1RU M !>JJD/5Y^S/W0A)>PPT$,)\_$%$8Y/N@6HWK=#A31%L?3NM;%%V\2,ARLUI M,&N/4L#::@50&5LME&[PYP24)\JW[%$VV[AA=,M>8\6-C$)#:QX69$@J4%'R M:X*)W]@U'1\S'+M8Y=!5V+2#X* [6I4!Z5D#=!%. 'E\;<&)[#I3%4N0[2EJ M1B8X9T>4+,5/)=L+-(F@9W6HJD(#)#'"R_#'@QU"9@V!=L/T;.1+H<@W[ 0J M.+)W^EE584PVH]H+U.*(-IZS+]F^(Q9@/ML"O+7&;?)JC=FH(;SN9(K::*O= M @7*49R%$\ L!K(&U#L/?5U"8$/)>IOZ !I9+"A9E]S9-5!@6^6^/>0U<-B MZ"*KA5,^;$CQ[/B'5<,:[(ZJFS5!X>.T-1=5B(N6!U>)=#_QK/[NSOI_<[/V M])IT+=H]*X9.&LF')4\>"8"L3Q];QJ)JK/.JF/(^N1Y.$DSZ.(8YP _C;NV4R M8G5S407AH# -"V<^!-1X#3NQ:%IPH)HV EA<,X]BZ$J7GK_(7#QV[B@-O)@0 M(_&Q@=K6V]'KW@H_+:[539NM[5,?I^2.5L?I.XO&[A:N8->&Z]W/K,>)_;9)/&4ZUF&;)0&')B7+>C8UA!IELBO'EA-^>DZ!9<> M]"BN=3%;*,:UUL-&U84*%=FSZX)##3;T/>J*& 4N:/%KBJ+66ZTV(L61S=7& M/5VG9,A>Q(\A(7C)G'0I+G4E*; MU*@.5J_.;.IQ# =)F]929'%\C[_/9K74Y,F@C-V!U:Z0"[JE+!/RWA=1>[BN M;^Z?:KL1HW#6Z4;&G--(;7RNYUH.*$!U(=N?9==>DZ0V[UJ*WX!V)ONLN'%( M%S&VQ#-G[7PVWU-^2\?::@&;N-K>9:)5^4GU=(WN\[BUMM,YH:3#T. M<;*(:+HK9%A.8%MT:T*FVTAO&CSC+&T6=*=Y?=B> 7^QUM;LH2R0?;KI*X8V MULB39-@CK'(@LI>R=-A83(V2[5IC?-4,'QF6'UH7T %J M-NAIZM2Z0 ^UIWI"9*9R7T.==1K 3=R6LM7;W B>.@F2J*3=_Q86V;L5':@H3'+I\7:6.0Y^OE]3A+7+DXFK9S)XF$'NAJ$?5HUU31_IZO-/> M9HEWMD?B;.[Q9LG*Y#T21^&,CEC$ZBWPJ5>N93RR;P^+L%(^Y2@UN :\H[FF M 1Z@NMLFG29V4=B)_%GM@EV_(O/>?7]&#:UI((F@T%YNLV@8N\;0(D M :8 30!!:K,T15JQ(XOJ/8FRKD'A; I@#H "J$%F5J6?&L!_X(+.NB-PC=,M M"E.QI4]D9BMV:\C7:M\2%ENX[=!;ZA'T[,?#;'123^NAM]59$B4NEE/'<"V2<65;]*7J+-13LH2K9.N M*\4 "E@;+",B LOF=,'R,J6 $UC2I7?S3XJ!?>WE:!\#2S:M2+2S"#M15942 M,EFEFT 2K!/ !#N?0L%RV%2PC$87[*[6!2NUA>2RV&2P1E$:['@S]MK(S9EE M*P6#A%R7*N\6O GQ \(R$G5 *M4(W$RUDIN$I8FF_$*YJ=)4K*S2WP?&E<(N M_3*BAD.H'WXS,FGU^W.,,GVK?EDV&HTS#$L:)!3697><05CE+!J6 ]=87<-B M,QNYC[N8:S\Q#LMNG,."!\>EEM'D[!G617O*P_Z-%P!_OCEF;BW68">(E<36 M;PNQ6,!#+)TV"@FNU4UV]"YWBDG$*B1V#HF?F\3Z8]V&$=!+K&MMR,: 9.B5 M:G5 _EO\7R@6H)M/(\6Z_@:R3MCY=?MZ6*Q:LA9K#CW;F>+O41B M+?MI+L)L(:P0-?>+9>B) >%I2+4"W0BT&)M*9>CR\F)VQ]AWXVRO(#2?0[ I M:5EL;8"Z31 M&^>9-L+5ZK7QS2K[K!0^(NF QLQ6J(P5LC?2Y0(6,5>MU,*VZNRN/TFUP],A M*,&6>]FV0*&@SDHM_.F&!K%\NMGY'R#/\,?QK-@Q_L:N0D*67>LOUOB+18"" M2(K6RR8B'9G 5N+<5PUN^3BM_X\FM98A.0;!W:I!2 M2X6TQLM\;;^QV@B];-U%7*>CNU8(6^\MFR@3P$^<;?^C1QST92=7&;>CS+(- M:BFXAEK@G*(6HFGK\_#%;-FY7$L.'NVU+)>G=3+N:4UW1$,VG> M&.-\1DKTH'_PGCNJW>B.2:5R*[=^K@DJK%=)6 M:X=X"+9)+@?5M$MF/:6F:I<]_-W!XL\UT'$0=!^N:U>J$-[8+;Q6/@NM9RUMUK!J+6VN$H\?;%-<>]WIO$%MXX&S7)2K@;;-A M>"NHAM"0;;R60*NTD]8:>/&U*MM&W;[V@NO-_-?F[NYHM=&)[CDUW6:P70B: MZ&I]L]F)3JKVOQCA9;=J7.<4.5XJXH7W4"/KR/+":S>VGMF&F^T&R^O W/ ^ M:]UH9CN;!N),4M3:NHR9D>W.^XJ8I.[[MPF[R-I'Z>S5VZ%03&] V:A?EW7PJ#Z45GT>D;="'#;BT/?LU;>^H:;LQK+.QDDNU#3>: M#T-]6-MVGM:V]9:Y;6DV 0FE:UN6 M$V6NOKP6'E27/_CX.Q*V8O5W=\A;8!;QJ_FZG?:6;D6O:$U"ZMEO=;MG*]K* M$[6A-1O#IN]P=DM3\5S.;\.(<\?D:J@V_EF]*AS-!JC_%5@HM+2[Z"(*&1S-<^)U^Q' I]/8:2!KZ]U=Z#+[/6=XB]Q:]I M;Z^M%-QO+8NWRUK2A=6==,6W%U8D;?E60VAR1=^J5]6W*8@NZ?<0!KC"M@/O!%" ^ MZ%2OG5<.P='GL/C C>#*;VVW;#0&:5F4^(?7T^!R<#VXB+N[KI[N^R%M;*K]W5$E+J^'R2&^X/M+2Y@YWTIG:_.'& M; B'0ESKJ[*F"NLDY54F<4^+[9OWS:RM;BN0G*;F;:U]$E&^[?KUU1GQV.+. M.KNX=-^LTV;1P9$J/)/N.H^+]M3KIIOT)DGQHY69_/:8"]@Y;@$42>>"7;8- M-)>%%-@][@46!OLN1;5N8'M_::!>X$C V+GPI?B]\E0 0+XW '=Q5RKO-.3J M1 66F\!3X!# "G""3<&N8"6YH-\7K'(@!OL8T$[2]^Z+E])-+G_Q!ELI!>6: MYG]RQYRH7@)O*34IN*K?',G&7Y.7J-\L>ADA0HHS3*A?:+?3^ M"LFWRMS4+"#68 B,\7]/ZUR25I H="VAW%6#4/6 M:,>Y_5/;G^0.G?NU)7[^.(%[C%BYK$!WMPLKA*0%6'5S>49.8[4Q=YO/%>1A M8H5IFEB9[\TWRT9?(Z?]5VV^!-TLFT$WXX::2\4BXB!I6U^DK& NR]B,//4#ZIG;)C*5 N]$J*G[[LN=MKXI'Z2 M(O^K^L;4K,?4(]N+S;*%9'5Z_KS[;#:%41,00\BB $,#_-F5+ V8S#?H>])I M$91:.^<+ \ET08"?6]]D3$%'NARZ[2+6IZ)@UK(+S=+3XVA?O_<_?F&7-V_\G[K/I-<#.+4,\F M@@-K4E/6*B68D1@ C@2C@#>@/+S4G=J5N-L)GLT)?RRQP-,X'BG81MB(_?CJ M[/)I%4=G\#46>4>QBUO,V-AVE5$^IHK6=KG;]=\" $>YOD\A7AYPH$N?]?^! M8X]T#+OOK_U6/QC$4[5!9B.9=YN/7(+V%ZP*=K,.@Q_!R]G.Z3'X'RRYB\M> MY5BF_SW?*)S.$:RCO>UQ@%&,?;MD\%%WS=?#"#9>6L%MJ3+,)W VL.D(4K13P2!CE;:[I+EVM#C[$J!-4VAA2S(&6 M)Y^>YTF#)E*0 5S+0P*SZ/B$G5HQL%_N'QB M@U.Q"MT)XI%PO7H.EALB&$6'PE)5H1/@"4 $" )0 4AJOD\W*S=S=UO'; Y- MAK.\"EG+[^7.!#PYM0(+T\*LID:QY:()N->BO:HV"_L+)KH*HALOP\7>0/X&YKB]3<3(3,-'U MAHVYR=X)L$S7#]IVY472=X7 %('*7^OPL2M519U"Y4X>VM/(Y9;V0\ST>V+V M^T!4WLK[3:% \:I']8[N=JN[OMW)J\#2">@ME9^^&1V\*;W$Z!CU RRAO=/J M,NEYSUT;,6".>:FV2:3>9TD#?1NFW]W0;?,+SJ1V1Z>[.J!-KW571FQ8"^XR M6]6CP3GO[IQW,CD"'A6N:ZBOFE3\,'LVDVHDGB=2=\F[^;3S;DSS['>I;>\N M6T^#FUJ8I[P43LL"+L[)@&_!F#M%G2[88B)+=6 B@FN\1=MLI%:8[-?;5=IJ M%=FTY%L69Z37$EGH!= ]@>^X,CII[RM$4)RU11.3#@6TD#@F+S\*(WGQ:WT+T6:J/#5\A;G7,83M?^&0I3[TRC.*AULB*E0.'T>7;-S>78>XET P96S\;KE!/9R>C'%H3C.[:9887NUT$/2;7:3QFNZ'6!J>ZFA[M7#XY]F>GL+31 MB=FK\5Z>[P&W,5B\[;P>;].U?4<3J6(SDKKGS.$Z-OV<+E+)I@(@>XKOS=3< MBTF/IL?LK2@5Z^SK]Y;\\7@1M8G)%" M#QFX1%\(KLR8@FOFTPXN".4),1V"7,NPH]IL')NB#MZL[=*Z[M8RUD@';N/- M2%JXU-O7+0S71BK#U6&V'G>I/DQZ:-H7]WJO6OO.0'VXP-(@[N.V$/CE\-MX M:8U[NV(HJ1(WRC:S0=[,;'B;TKTMYI64FFHLWMM>4\>86=SD9NLF(SI^3/Q^ M<:_%H0'I)IJ4C"N C4F61)>+4D18K0I0-LS&!7:Z<2=W<%S,KQP7DDM&W.#9 MWG3?+?1P+<2/'M]S$KRZW*FK_O8I>_7ZYH+TO+$DW#41AM?-^ M''&9"5[R,LFS).!\V@E8-^ !QL)1 M"%]_$E?Z[J=1@MS2>P&/TF[&_UQ/+%N09SR*_0NI7;&^ES=_PBIV7(Q;5>Q* MB+N,A.$U[*&1+GO172SP%>]U3N"I;8,63CC6+?55=<=SYTQ>GOLSN*K2)4/" M#9QS+MTS9 '5WM@<-)WV!,8"-UV[GMPS!;K3/0KV=!=^]E?['G<4J6;LO;L9 M=5&3.E8JLONS8W@TY1'"[9:]3MJI;E,7-ZIJ!;(V![5R_RZ M(96RN4+'+J1W^)<-WMA1=E^'?%.M[)QP-3P"QL>J@O_ ?S5'I$!QL#@(EOF4 M%:>(?^)U[2+X/M<(3MM%@#?(D^!=H2/X<+KI>Y F"6W![6!(6C,8\CG,4YR MJ%1^HF"OWS1X#?R_305C@]-SK. GWZ!MW\A&BP7;=V?!]5D,'/<-OLIB8(=L/%@8K@XG!VUQC, ?YQ.IG$YMV&0)I,<)M3^@VN-Y<)&U'OR_E1LZ5>XV%$A?<(VWGQQF_">;82'! F$.,D&XVGCI.PA[ M@A?(JT&&<*_P "?)QUC@7J-8!RDG<#+F M5MV1A+L^G<]218ODHZEA2Q.S-.':98;1D=!6YF8J_R3 -5G%28EN::<*ZAL] M?.9J+L@;@(8 .XRO&Q"F5F_"?N-10N-/LX(,.& ?OQ_4ZQ> M8EVMR3HL';B->:HL:)=V"7(4-L;Y0%!(?-#FY8@QY,OGUU%@= M=GM]XYV[K'/0?)<#VI@^_R004D#SG\;J?=Q?Q07VYO"?JE L8'.51>MH>D9( M:%.-DT;5'U93M6QP_%IJ*-'*J($UP/MXK?SC_$+&7]%T436 <+QNL_R,K1_; MA=2%8U;X,679!F!9)J4)/WF70[8FGXAVOT==]CVP8YFQ%64U7M(T-]"_.Z&]E-8Y^5#QEXC]66W6'?XM8IPA.])[_ZUS,'#WA(#;2C=,%*2 M"IJ$*341XVAW'5MI:K[EYK">_S=8HD/NXOND/1A^/6/,R\(6@ YO50)9U0Y> M_.0YMW%-[M]W 'J84/\U\-/VGWO#7> MB9G,>,>(A^2F3M,?CM7R#>.N&;OS7-[4.UN;5<]Y:+O,?+L?ZFM590QF5A^R M.>@TH %GJ']8F?S1%2$&!O:5,37L\H#OLV:5:Q07+M]U0H8W?1&TL(0X&R%P.6Q5/61"88R8P >-------------------------------Cut-Here-------------------------------< ---------------------------------------------------------------------- / e||) Lyndon J Clarke Edinburgh Parallel Computing Centre e||) \ \ c||c Tel: 031 650 5021 Email: lyndon@epcc.edinburgh.ac.uk c||c / ---------------------------------------------------------------------- From owner-mpi-comm@CS.UTK.EDU Tue Mar 2 15:16:27 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA09733; Tue, 2 Mar 93 15:16:27 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA24112; Tue, 2 Mar 93 15:14:04 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Tue, 2 Mar 1993 15:14:02 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from antares.mcs.anl.gov by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA24104; Tue, 2 Mar 93 15:13:57 -0500 Received: from godzilla.mcs.anl.gov by antares.mcs.anl.gov with SMTP id AA26860 (5.65c/IDA-1.4.4 for ); Tue, 2 Mar 1993 14:13:54 -0600 From: William Gropp Received: by godzilla.mcs.anl.gov (4.1/GeneV4) id AA08480; Tue, 2 Mar 93 14:13:52 CST Date: Tue, 2 Mar 93 14:13:52 CST Message-Id: <9303022013.AA08480@godzilla.mcs.anl.gov> To: mpi-comm@cs.utk.edu Subject: A proposal for buffer descriptors The following postscript file contains a more detailed account of the buffer descriptors that we discussed at the last meeting. These allow one to send non-contiguous structures of mixed typed data between heterogeneous machines. Bill and Rusty %!PS-Adobe-2.0 %%Creator: dvips, version 5.4 (C) 1986-90 Radical Eye Software %%Title: bd.dvi %%Pages: 8 1 %%BoundingBox: 0 0 612 792 %%EndComments %%BeginProcSet: tex.pro /TeXDict 200 dict def TeXDict begin /N /def load def /B{bind def}N /S /exch load def /X{S N}B /TR /translate load N /isls false N /vsize 10 N /@rigin{ isls{[0 1 -1 0 0 0]concat}if 72 Resolution div 72 VResolution div neg scale Resolution VResolution vsize neg mul TR}B /@letter{/vsize 10 N}B /@landscape{ /isls true N /vsize -1 N}B /@a4{/vsize 10.6929133858 N}B /@a3{/vsize 15.5531 N }B /@ledger{/vsize 16 N}B /@legal{/vsize 13 N}B /@manualfeed{statusdict /manualfeed true put}B /@copies{/#copies X}B /FMat[1 0 0 -1 0 0]N /FBB[0 0 0 0 ]N /df{/sf 1 N /fntrx FMat N df-tail}B /dfs{div /sf X /fntrx[sf 0 0 sf neg 0 0 ]N df-tail}B /df-tail{/nn 8 dict N nn begin /FontType 3 N /FontMatrix fntrx N /FontBBox FBB N string /base X array /BitMaps X /BuildChar{CharBuilder}N /Encoding IE N end dup{/foo setfont}2 array copy cvx N load 0 nn put /ctr 0 N[ }B /E{pop nn dup definefont setfont}B /ch-image{ch-data dup type /stringtype ne{ctr get /ctr ctr 1 add N}if}B /ch-width{ch-data dup length 5 sub get}B /ch-height{ch-data dup length 4 sub get}B /ch-xoff{128 ch-data dup length 3 sub get sub}B /ch-yoff{ch-data dup length 2 sub get 127 sub}B /ch-dx{ch-data dup length 1 sub get}B /ctr 0 N /CharBuilder{save 3 1 roll S dup /base get 2 index get S /BitMaps get S get /ch-data X pop /ctr 0 N ch-dx 0 ch-xoff ch-yoff ch-height sub ch-xoff ch-width add ch-yoff setcachedevice ch-width ch-height true[1 0 0 -1 -.1 ch-xoff sub ch-yoff .1 add]{ch-image}imagemask restore}B /D{ /cc X dup type /

FK4MK MV^JW$G9V0.#:M*:M,VNK@+4>KE%KXAJT>JV-J[ZPJU0*JMB%QRM M@HS-B+XBHV>%\CK>,:\HC/SZOM*OW,;."KBVKS:C A :D*\,BOF*OKZ<]>O. M>K\"L()>G5>S"JZTGNWZO\:O#6P8@;ZZ!;0>\(JUXJ_AS /KN^:O%6RA>> 1 M%:G%1'H)-A60('#A$_@%Q!\=<$DP""C!);%I6 1M0$V0!M!U&<&VFAC0=?9! M8D#\L0%UQ!J1$:RL=80OL49\K,R@5) Z@!Q)*K:3' JM/J.WF.\1HF9#-0G% M^A/M(FMH30HK_ 44NQ*TBTXL[?D*_15>[!)K-F"A*6I#<+7B#E*L-/JQS 5/ M+,*:A!0E&0<<"SPZD5?K@0#&@I#I28]YM8HB6>R[*3[B 'DL'0NG=9'.) TP M=XJGR"G=-Q:,?B1L/7&EF,6;(?F=%(T&[!*,Y"I<1!):( MEJIA& 80?O2C$AOJJ;)X10UQ4N 5.!YY&J*,CF=#+.MTU*,?;#\JRY:@)0>. MEX2D#JULU(I7F(N^+!M+S$*OQND\:1(*-LDI*AC)Z@1(0+=:R=X!]*2@M\KJ ML<'L&P'+^G@F(1)T3M2RW*S9P,?FLC->-RO(@K/]:&\#S,H?-8@VR\X>L^FL M'@O/+J+)K+&WS-H,OJ8:"LFBL//!8'B7AA&I+!N[/XZQ\R4JZG3HI:Z73X8>>E@VM-%J%LH8PI$5;2[J>!VUPV5YFM!.M=-EA.K2S M1&.ISK:JUFPT2D[>EQ>JV6!*AIF$Y+)02_ZF5*EX^5NREP#"ZW@LUI*6*=+G M4D*/\R7F1CM&L9ZH$[N#SK*T9[X7T8IOQ*,J:T.^EH^BG%AZ(J=5*M[WS$H! MT:P\@REJBMOE6%@-GAMG86]3E'AY)RV.!YZD#BKMC)G3!K4O[1D[.LJT-J1) MB#4>)U!MG%@H#J%4K1JJ5-XX6EP-<2]\M69A5.#,HK ZK-/PSP84X$:H-\>V MJ@M!T&D]!I<.IVH)13XX&6!14ERBL:VHFGG(JI;.9$D@UI:1,FTTVM-:HQ>M M. FKU7L[+6"I@LYPOF5C>X)ZLVX!47MDMH^P): XU8H=5:VE*,E&LWZ"'. G M'GH"K3ZYV.)X>H(1F=M>M;SM1?#;GK1D[1=+W-H,QJT^^_P]LSL*]+$&Q'V? M25(@.< +[.IJA[)NMVA!-W'@5;=""N6@BQ!\>PAI)YZFLM=)W=?'E(YD*?0' MWKH*;0AK2*<)E/.ML4!^)I/.))21W[8AT&-)NV/RDXSD?QL45); &_%HX*:0 M5>C]Z-\2?,9"]JF(YIZ](PBPX,:5DJJ%"^$"N&3DJYD3++C0W[M9G"J=:"AK MB..HH0VJ9C$VDG;=XR&)3CXX*^M\NXPD@O6$E1K-6@3/RD(0$K"73VB,2_#- MN-9M?8CW7:EY*ADP%9B.L^K3.6W*#P#$CTO>UKA#;C3;LM*P?213F9_2CM\M MP??D"KG0;,G:1X8$J.L3FN7""UON\X?5N@K^@Y&KG(*3S:79Z9WX$O-MF8OW MG;E3KFXPV%Z*6D:4^5O.ESDM!ON&CJ0^;14J8N*TV^>K2>A"E^"G%MI+WH^" M;E[)DV*;9.5=.::H21N(TIN2K=6+0I[YFJU 6N%ZJAAFS5;O2=(NHFX[N099B*> M+BVN"^PZL08H"IGKSI?.917*ZTJZ]6B8*4[JN*-C*IGK,KNC8S0ZT4*[0:UI M6U*&FK=N4#MYCKANI[%G9F@T(JR&)^?!'W*?9Z ]YHP[8Y3)7."X4B64$2/4 MAT!;=ULA2 $MZ\WP2'JMCIZQ4>3]IPV"G)HTR@'N9*NH+%8()PD.D=ZFNZ$> M%))>G(R:BL^Z;B2\?LLUOSJKPD[]%1IZ&\4@#2F^\ED38O"C %2+U[&LH+ M]#:]3^?FR/.&O!POR]L^@)9?;]'K%W*]OP*,:_.B-D_ RIOO=JCX9M5K!+B] M[,JAJO2B=2DOW?L2Y'L]9]5+!.B]Z4G9Z_-NO?ANW1O-H;PXKTZ0[X:X=R\* M( 0 OE)EXQOS*K[LRJ0&]4*^:D[C>P1 OE(G+KO[@9JYT:SJ>V6VH,CKIO+2JL>JO$@0DK*GZK\D$/"["&JU*?93@" M?A91PFHP98 (@!&* +4!!1@+EAI'@AW1(E(5!P&?)P"\!&^A-TBY9!8&I;; MY1O!CM".9*L^.?^.CO4O4-%!8I1]*V.+FMIN]=X;P3M>E@'PKQ!1]K_A[4=J M'CXX!K"#,SI&E 9PE#8Y;I7'1 1:ZKZGYT0%W)'2::F'%WD<)L!EPHXK+-B_ M(J:;V0/>CAAH -Q,>I/OI@W!.W*: 3"I]CK&P"=%POENUL!JSO084TFP0%D#.\%-8M3:04K!J4C^NY0:JL-* MK2F4AH\2\%**49ZBO&S[\UE2D7NI"?QJ H^D@0:*0DZZFRFG"0W:ESIPA]D# M!Z5[<'1I!%>@'&4AF5GRCN(E%7P%&\+'G"8<7=".83"2*0=$M7>M'@D"XWIG M\+V(9[AZ>-_T2RH6MELMA6FNMI*I9AMLZ9IQH8$2.0OSCG8F,@?8JJJ+RC":Q_[J7QJ""[&[VF4:50&L+ !2SDZ6B>T^4%3NI#.J=6@N&F+]K.20&9M]TP!D("Z'='A!*&@L+S@![4N#$Q5Q]L>+:>45+4""=]@;= MXBA(W%1\6[&\$&7!.IX,8^GFY<&X?"M"W2*&X& MJ_U+XN%'_!KF+4BX\*8DD: ;,G^TJWC?KE@4H(B!(NG)+H*#0&(*2]&)Q=YB M<+CD%7/EA77X#DJ,JS#"BA CO^UP2.@5_W[^H7:L7,2_OFHU"![KFL4<]+?, MS'4'L=*I'CL37_$SZ!Y;"O!QVUD?=\?L<)6@"OL9!]YKJ1"'A-AQI?@?<\5@ M1GCLUPW(X7%;2AXCR!?KX8>+/AK@*.?R6L:6CP;ZIR4"B6%Q@(PD6HF&17QL M(-_'%^L0T!2"@:#@&M -A@L50GX\\"[(JNLYF!V7R&#&B%PI\LCRP4Y\, [) M&J&$?!90R#;RA1PX\'4FH=VR(C*#Y_'TVBGZR.+@@\P=A\>.77T<'LN, S*+ M/.0UR= O6C 9L+\LHA@LU8K(\N"/W!P'R(.BG)@B%\COQ(&\)&?(OZ@0<'S4 MR**PDJF,](!4,I\()(?'K-YC,!_#R4IR]G>Q3@:7(F$@L'[(MBV?G"97R7_R MF @H#LH6LIQL*'_))8!I>)'4 73(:Y 1- %9'^3']I5]5!_:QQOL@T$ 9F Q M/,@ZP.CW I@!=H$"DB%,!"#"16 XB!^A\M;7,.@6J\7H%RU^#.#Q;K#^>8LH M0/TP!^#*:Q_7]]6A !*BU[BGCC/R0]4 ,K0/P#*.80>>!;[RP2A 3 6SA80R M+%<:$F.AG"UOA,)R/2$P@,K)\JY\WI4>K_)K(/LARZ(RUV>6T!E<0IR<63PI MV/*GR"ENRQ0!<.$MUQ/5,I'B+0;,$V.7,"Y[BQ:"V5?UV8+2B']'@[3+?:,' M0@LNS%??BW O@HPFH8X8!L@31<$5@#J@!5MQJ_PJQ\JOP:P,?> 3MS*Z'"_O MRG,)^_$K]R3!\H2R!Q3+K\&QK#+KRF' LMPLPQO3,;3,,VLJU?*]? M$\RK<-.<,)?*#'/4C-Y1,!"S5R Q*\RF,J- A%S,3#+>=^YB!FZ#7>"RF<2? M6M"@5N@@AZ /PH,$:OE!"9=-L8XE@2,;\+*-ZL?;6.1!?V'Q>MPE2!L1&"FSG4G3@P[;\AVY]X),'JG ./SAQB( 77 &: BC _:24:@-<(AM0*A8KS?:>@,%.RK &,DFSM'QVM![_P[(Z7"\]Z2-4R1F,CR&-NV MMW;*6/+,,L^;(OG,^"Z_YRK^R>6.SJ6B8?M@1IGD;N+1V&K&#N3@W! 4SN!S MP#MWFH10@$<7_UF#(X(Q6!,TR.)@3> XGP4U09^L'DQU]T-H@2UPOQ*J@G & M=G\B -N("SH:"LH:\6N$QPC?L@P=<3:!E1SRA[ M2-#WMLRE!6O&$( $) &@<@H0MXD!1F\*D%BB""A )*A&VP5L]''C9?R\H6%S M ^9M*@!)W,*4"; T5Y!0J<#I "2&EM@!*@!*( 9C49# M 8:T6\!&+P%4 "6]R9F\9<8?'4AST71 "H &H ")=,C'2-/1:0$D+4FGT4() M&WWT!I9PM!Q=)AQZ*( =;0;@T !(1K(TP' 2G +2T$@!B.00J &='29@1'!WQ3 PI@(C"1I:GE 4NU1 M,]6 0TC]5&_3)35JU0U'1U6 Q!*=8#*&S0,#TP*< :$U7B!(1W-L=$Z M-5;M44L*' ືU1##KG8&.-:HS5H=5V/4X/1M9UC])L!2[/1T(?&5]"Q 94T+7T26!NHB!825V?71]S[FD\' < % M&J >"--OQIHA(;X!<;17MV8 V()C"N &W*N0 5@ 6P\!]9T.0E_3<6Q&!8%@ M[]>@X&E0:1('FS47D/)6=5Q X2A,)P&\P1WM%ZX9SL9G_5C;V#, ?CU*=P=D MX5F08TO4>@A*<,CU!1E!WA:H"0N9W$=YQG+7C_2-#7,YTYET-#U)4]/6-#:= M FC3W+3*.UG3U;]U518@U-$2M94=K_%IV$X-XBADUU-0$&$$C-(+]AD"&XS3 MI1IH$*F]@*HW5@ M4&:PT;^U!.?@0!E_-"2]'S#2!+75FV:LL)(U'!T$]!-^05S]6$,!/$U)]89]KV* M::8 J1N]]FN[VP*WM:UO']S"M@)0<-_; [?!77++VQZWR9URH]S%-FJC-------------------------------Cut-Here-------------------------------< ---------------------------------------------------------------------- / e||) Lyndon J Clarke Edinburgh Parallel Computing Centre e||) \ \ c||c Tel: 031 650 5021 Email: lyndon@epcc.edinburgh.ac.uk c||c / ---------------------------------------------------------------------- From owner-mpi-comm@CS.UTK.EDU Thu Feb 25 12:19:23 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA26006; Thu, 25 Feb 93 12:19:23 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA06312; Thu, 25 Feb 93 12:17:23 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Thu, 25 Feb 1993 12:17:21 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from daedalus.epcc.ed.ac.uk by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA06274; Thu, 25 Feb 93 12:16:42 -0500 From: lyndon@epcc.ed.ac.uk Date: Thu, 25 Feb 93 17:16:34 GMT Message-Id: <2153.9302251716@subnode.epcc.ed.ac.uk> To: mpi-comm@cs.utk.edu Subject: background information Dear MPI Colleagues I found the February meeting both enjoyable and stimulating. It seems to me that the question of process groups and communication contexts remains fairly open, and the committee has taken steps to move forward. I am sending the interface specification for the system which we have implemented in Edinburgh, called CHIMP. It doesn't contain heterogeneous support and the such, but it has a very flexible approach to process groups/contexts. We've used this extensively for both SPMD and MIMD application and library development. It's coming in three parts, first below, and it's a uuencoded compressed PostScript file. >-------------------------------Cut-Here-------------------------------< begin 644 interface.ps.Z M'YV0)4) F=(B")DW8LJTD.$"AH(2)8;(*1.&SALY.D"0L9,&SAP0-5S0R%$# MQ) WBP*9/QX!@7&],XE4B13IHW;HA4I KB2ADR((R4$0,3!@@8.73(B$LC MALT<,YQ""7.FS)R,-/)"W-OWB1PR93"""#)G3!DW9)P*>5,'C6 M;8T<-D#@H"'#*9$W8^JT>4S'"%@Z?T'L95/Q\IL64MZT">-&-M#:;FY/9H-V M-O#;2>@ ]>FT".23;5:[@?V0B)4D Z'OALSD,MFL'C6BQLJQ^O6!1^O(<9P1 M!)4R6$!0I@.GSLN8>!LRC %#!UP9.$A6QAF702$':E.404=&=)2!APMP'/A0 M" J\\!X61/CTD@PUN-73&"\A9H9[\&4((@@)$=C;"T[L(6(?(H+P@A![B'$9 M6B\Z(>,4>S@X!AI]Z/@"%GM, 8(3?2@@A(Q42+$''7+P-@=M#08I8QI3?F0& M4',4):0=@:51&HZZ,2 (C0%55)QSNB7I M'(@J"I8"8(I95*&-@I!F4$T*.N*HI5;:6Z23FMI;JF."T *KF,($ D]HO>J> M% KL!F4:>("0FAP33:?L2LV240<<&ET+ @T@]/72@96AQ:V/:(!07XC:6HMM M2=Z" "YD((% K@+G7?2RY[W9U &5& M&6?M,8=R=-0QQXJ*.6*=XF8RN34=%'@"#,,.18+\F\[)X*&#&='(TN_-K/;_1+,]+ZCCQ M2IC)*$8878(0'U-1YB&C$%')'%Y\,]91-1E#H,'4'@I$SI00CA.7F+[.^=E3 MWTFC_1B.UPILQAMOV$O'Z=/UD1/A81A>LNQVR T"&V_P9#/1]-HG8]-+;_%R MU +/,:)=0KX-Y=UKHQWUU&%4/;P9<[;Z@O&"RZA\W%M@[Y!;V&MZ!M!"/T\U M&R\7L0<<*(& M;H:E6&&=ZR_U.7JK[W\8PMWI$$&'6C8P_[(4)$P* !^4\$, M .,UASJPQ5OZ0T,+T% &EKA$@!(DH'*RA:T$GF&!W&K@ Q44P1;@ 2EFV$,, M -2G#!:0@[=[S ?+=381=DM!G')@"?. 0@RV0(-A@*$'%Y@3&[9KA3=0@ A+ M2 8\^!"(0I3A NUB1!(J;']IV$U?GOA"^-&!;&6HT-XN\T$PNB]@P&O7"X ' M/+L0ZTBR>IG_GO8".=R!CNNAXQFP9;0*C8&/:'O!$>CHO4!:3@Z8>UQB)!8& M.Q3E;'8Y$!N"8B3XO>!O@6M73F[DH!O200%&FE'BPA">=AE)C0-\87S8AZTU M0NEI VR66_9W0C.,:'\\M*6S)$A!"W[2AK1$X2[YY[\%OA&7PKS?3WYTEC)P MQ#'#[-__RK6_7K;D)1^C&= 4@"=?N< NP=0E,"68RQ%]JXA.'=P M0XV6BH:F+O0%4T@"IQKI)1"TB26748">VO &1UI$1E9P546HY2SU1(L.^*J6 MMNS2+O@9BUPVJZNVC$4L!;C@)K>C@\"B8(4^"$P*A1U4&>14%-SI+D:L-!=* M]%6&HF;5GA.[2%'F4-4[P,&=(#CJ8314H0*OD6BH^_(&!AX+U M NLD08O+#1V'0@,[P^G(3B#@4 T4L(?0&8F*!1T?_-QR-DXVJUYCL$-O0G8Z MI]F@!C? 00Q<<(/06(]718G/??.[W_X>D+IH"%.O7"8D.$BLL_J2PMJTJ:=N M!DU(0CC-AFP OO0&4@YJZF0?ZQABPQE-1G80&(BG8CC&K;B3\2FLPJRPAYZ P4G0KG(^AC(Z.H@21]UPSVXY0:>O$R#Y& 'H' 0 M!1F: QQHDX<4E"$.\L+#CVQ& P5X2\J)J3(;4. $^%#!RV"^B&$CNR7&RFJQ M73)LG?.L6,;*"S)[V,,9YL#58WGS3M]40).$1<6?3,5=(6[6BPVWYZ)(N")[ MR/ ; M7.OLRSGF>82&;S:5A!$]J1AFY!HE5MET6_6-(EYI2CR22L2NMD;9G6 M,#S=^>ENB1J?[[RSGQ5&6,H-NM!0DE)]V^ NRL +K^]""UZC$B4W&.\BS":K M65/W:IO)@0W>.:M#)GW&\7T[W-R.=+F]#6Y-G75^D_3U/37[LC#L0=L*>L/P M%KO!$9_/1=IZ 6+8L,'XN$4.^+;(RYI0)'/%C^!!?./YAMIP;$W\BGMH ;>: MD"2%D2'C9^.XP@*V$!"(7'L9M\O)7W &)YO\90%4N1SWD).5JV$/(7_9&O:P M\815Z XN1WA9\_TRJV%+XR"XP\O:L#Y?G4WI"I/JT7,"=1F]H>G=K/H+XM!T MNVA=#D4.;])?-B=L/?UE@L46M[3N1+<82>C;?ED><"YV208%6V$8ZAOFA-6. M01"$@J(HDND(10- MZ&MU3.'\(%._*"JPI#?5VDMXK$(0N!=L22#"&<7NW@I5P0UIB,/!8C][WGS\ M1>MK'RN%G>?Z H4-VGR]6T+33;N'N0S^= M>Q]M\1 7EN+52;G-U=2E'&4M\ MY'M!$Q2TDC%\!*\QVO'GW:" SOV8C#T2<[F\:,:!36= B1%H41!JY/:9E"WK M%4/!!P>&)3Q]EF>4$S87M5;,DEIR(C,P4 ,P:,R 9$(#]AH":P(3 )!@(VL$*!!"8*< ,X$4A5 M14=VD#HG]@()YE.OV!C"(B2\,3ZY. 1,@!UXU#\)\A@@8&M'4T=HD ;)V!O, M*"-300=35F-"$AP-@HTR$@8TP(TO,!5G !38^#) T"E341E1T18C!@2[Z"E[ M<(ZSIHX5Z"BZZ(LNHS! ":S%H_H6 ;TV'THAH_FF&"JHHHBR#C$:(S(8XX M)C!@LEP+"06_:(YC &X.]@(3*7:!-S)594OWHXJIDX\5LH\H%)(O<(N"8X[! M.!4"$XPKJ8]VE 8J-%WV.)/+-9/16# @AC R^8SDQ1\"Z8QID)//N),? T?Z M.$E.%(YLT"SY.#*3-'=.V6+FJ!Y-B94Q69+J095=&9,RHD7_R),'@X)_AA;[ M(C.'2 =5X&"UU7D:@Q:[U5MO"2(5T@9HT2[Z!S4:LI=?9E1=,EHG0HEX&4.G MUQ:I5YB[@0=#5"Y;%)?)0CI*=I@RX!9OE%2S@X(+=8*%28GAJ"#7J"_:&##Z M(H[DJ #Z\F-/@@?L$S@UTP9T,(1;@HIQI1')9FW+9A3#HAQN\ *?E3V:XED5 M04T8^5D E"Q,L7.W>6T(!R?+F9L)IV^&]9S8YIS5QIS,UFY$9UC,EQ-V%U#7 MF9O=>8-!09T(QXJ^XIW6QY/*5IW4UI[G&1KJ.4E>:)[,)I_DZ7KVB9[S&13/ M$G>E=I%[5P;&"21]4* :R 8@50;"N4!^4AG6J!O).2S0$CO#,FI%P7+^A =1 M(UG8A+J<3).HSRNYP:.!%QF M0!LMMQ=2)@2T,0:O-2CZ$INM&3!P@ LE!X@*78D@=F.BR %EE3L*#CXZ52 !,X<'";-A;" MTDU.&B$UV"Y/ZC1TA4/@9T/@UZ=BQZ=ZZA#M$J@G!7ZQ$E-EBI$@@*:0JHUJ M^G%LNFYPZJ%R2J<4 S)ZFA>&ZJ>>M&Z"NBENT:@C(JD= 0*/NJJ4ZGOKHZ=_ M^A*#>JB>Y*;'$D=[<*48.2=?*GSM(P,WP*EV&J:R-J:.FJ:JBG> ]JMP0"_M MDP/$VB!W*J:/1JAG$ZK"DJB;4JKC0Z@YH:UNX2VXZJVYFJR0NJSNLVF5N@?. MJJG2VFR=:BZ?.JJ$.JOENJBF>JZIFJ:MBBVOVJQQVG_8(JS3ZJE.$ZXX1*B( M"JC=JJ_"J)NH2GR;-6MBD >@$:M.(Z6Z$UGJD2=]5E]82[,]FOTIB.C$T=$FHW8QB7]N'VV M\B@C&RS8YUSL!RR9TJVSIE:#UH_6A5U0.&L+!3X*$FJQHR\>D11S8B0#IE_\ M90,'YBA((B2R!5SO@0=&P!("$[8D, 5<1093,*^GUE5U^1*+V9B9R&-WR1=C M:6LQ$AP(9[2>TE;0PAKL(V6&YIO&0P)?&;:P(2-TJY66*X*2:P8D,)7F8CR7 MV[F?^Y2A*[:8.[F&L[FIZ[GO);H?X8*.^P:02[ICT&(50K=C )6M2P)S\+JN M>[J4&S?"RY0Y!(#!TF"3.P>KJ[K"VY7T,KF@:T.MHB-T^[O"R[S'>JV3N[O9 M6RW!0K?&:T.ZFP?1Z[G:^U*3^Y4VM&C7D[/N!(Z) 3@$&K\TBQ)D8[\-5H,B MH@! H):3UY;Z8A48N3[2:[KBRP;-Z[F:N[YR(#LD.!V/.QW\29X+VJ#EPBV9 MR2W]Z2I#=U;,9[+Q9B3:>58DC&ZZ6<*I(Z!=4J!],JD,6J >''=",CIO&[.A+?SZK/YQ"8\IHBB>)J;.(GKMR.A.(H\O%F%IB/'AFJ(NQMR,K6R M*\&T2\%-XK^@N(A 48JS27"QZVP9@QIDO%QD\* 4K,9T@#&R):1P%%L*0L2C MN <3J0!V\9?!*<-C]<&ILXH'I\(V$Y'GYFY__!'@%\@HS,+U>YP7V1$+58LJ MN6B[" +\N+U% 9,'\L/("#%2A9-:.;XGU2IR<)2>3#FE7)1?.;VCC"F&I2X4 MB\FW8[K@![J,*@6&EI'%H//,OF2[T6^+)3:[RUK,#K-K$8:!+%2)$Y MH<0:/QC@JH3N,@P=1@A:,L[>U&540,S,*/03J'0/LO=[M_=[N'=_T#=\.,=_VC=_N MC=[H/:R?/=[G+5W=#0-%\-TW4 0Y< ,Q, ,^0 ,X$%0\8 02/N%S.N$4+ET8 MGN%BE>$_N$@;N$27N$B/J$#+ETMSN$][MT[CN%!_N,\/J<8;N1$CN0;KN1"?N1)/N31/ MON11#N4^+N5!+N7>K> V< -$@!=X[.+8'51" 3;^1 .B+0,XKMV??0-#<. ' M/@/J7> %'N P( 2R.!HQ8 ,-7F9$P ,W,.'29>$D/N(@WNB.#@.* M#ND7?N@2[N/\X><(7B9H^^)V4>C2)>4QH.=)+NH^+@09'@.FCN$S( 14ONJM MSNJJ_MU$[MZS/@2U?MK<3>L"KNL$WNN?[NN2/N02ON.A7NGKS=T^;NO#>NS+ M?NBA;NL2'NKJ[>Q&8.MX/ ,Q<>@R$>@O7A*%3NE(/N$[3NFB3NOEKNPF;NN? M?>SK3NOMCN[,[N-&L*+E/NSR;@3$/@2AGNSAG>[A;>NT'O /_#'+O#L3O#= M3>XD3N\"+N&V[MTRD.U&L.V"G@3;'>JRSMT\$^Z,ON]&,*ZD?MIX$;_ ?%?7Q_>NZ7O6]?O7'GO4/C_5>7_!13_ A MS^P*X.\?K_'&+NY+C^0+'A.@7?0^8/'D#NT3KNQU#_9X+_5#(%8&G^0_O]ZZ M[MZ![_#W_O#%;OCZCOB!__=:W_A[;_5=O_73KO86/OGVWO8)/O%P;_&5G_1* MW_F._^MX'/FA[_>GGOC)/OC57OBL#_B*C_BGK^&\OO7\;OHQ?_:)OOJY3^68 MK^V;O]UT;MI+?_OE7N#S?O+0?O)SVNQ\/O/++O,P_^LOW_+A??:3;^OJ/0-G M+^I)U/S,7_,5COP[+^3[GOND?NC+'MY%L.PK-/04'_='_^R2+@1*?^AXO/JT MSOXY'^_\G_].__] C^DAN@!X[F:>_UMO8D7_&<#]=P ;( /I_FXW:<;]A)N!55!-3>NJ-Y(/ #BD D-P*5GPD,@20P!9[ $DCSQ K*TX P M\/-ENS#W^PA?[OMY-I#VZ<#9QP/_WNB+?;:O]@G!'$ R&-R& QV#=JA.# M9] ,]@PSF .$P,AK@V_0#8(\.#@'C8#-*P)WT.;1/#WXV5S@S=N#?S +7L%! M* BCX.\[X29L+UM@G=6R=D M;_IO_24[4;C>OERR,X7K[0;H/U5XVMS;#-!_KS#9Q4+ U^R,0/F3<#F ^^FY M^$-[HD[[*;T :/C*1((C>ACPZ.$^M:?Q&-ZQ.WONSM^- M/;V7]\2>V#-[)FX %CMBA_3I_N"W3]$BCZN%_J^=%@0 M"^*(8XEZKB#2P\$8]NHA C2,A!$Q%L;$V/>R'Y)SC'PN_241H[@(#2+;JP%N M#R[40&6G!0DA9[2( -#=^;@_N.X4 !\DC>GM#=K!.A@'VQT<]'&OT;O%QIC@ M!F%C;92-MS$F++A6^ 53':13>4ORU&_*S?J,ISV\XZP3L"%QUSW\Q3Q?UH'F_=KOMYKD[5C$-\ED1WW\4!DVQ-6F8[;);CD*.64HA5DD'$101(X$/GIC)U8 M>8@M\N&)NA+IZWYBB.R)2/)(*DDBJ>>,7)#\'@^Q Y*Z&V?X8.+;2X<-TN'M MPY#7XKK;)@1VZA$H"KOOX?P2'H@T?N[Q['U'_H!$PIRF@W_"D"YZ1YF7_IH? MD=-P,^_?I3\)Q_X&()?DACQQYAG)0=G\"*6@]'Y0LOF%0WU7[WJ>BN1^<:Z] MP<2]:/34W^X[?%QR_27#9H"O%(;';BL81 M3@:_AX?^6"2]BV^>,"I&0VF9#:>EX,.!N*_8)3DA,.2B/S(Y(["T?NV&2*"W.KTNCEP)E9%_G>KAR( MP;)7XDQ@R3._A\X4ECTST@G-"R?TSN'[XP\!HM!-O"'P CL@W;-W(2^_9#\\ MYQB'5=4TF)$0:UK-?KBAP"9A,/XKG,MD) 6[V< M.YA^\-7MXXBZ+U,6PVY8**]AM;R M9)%2:!Q)'Y4BWB MQ=PWY(:G=ZL+L*YTRLSCER8K(Y\DEGH.";9++#W*!+?P=/B.:3H\2HS=/UU( H< 5"P")')( M?KP65_2I+[+D]O6<"I(MI;P!:N *:)[UG WV@=1*">K<[61J1 MW0?T;T@NOZ2_RBCODB+;*Q.J<.*U(J.'"+.NB7Y+"=L)IX]=(^SK^O2"&K(R#\HJ>QW^%!$Y<+A5],L(.V MT8P2 3]W"=FH=UNCE="-_L8=]R.!8T:THN23A +10Z?][)[\^XWED5]"T>HW M1:G?/W1R!'*0W@!^V43/XA!5A/"Q N)1+,H7@>@&S(6,$.$=R/0W'/DCL&ND MC31$ <@<,$IYWX!4$D-1YJE2'1G]F"@I)7)"H#SB/D*JZAA?R?)SE-3HB<,J M>/?FGO-;?DL1V3DY?CGENJ.JTW)4;F!>.:C'[,S>W;-P3Q/_1;PKZD//',R( M##6@)*R0-K?=KF+M$WY[3KJ<3!&X/=,?5#R78O)W KNP*5W.J614=:J.[?V\ MGW=.P2FO>WA!( 80@1L0!&Y";FMP#J'043EZ"D]'HQ <X)3%LA056 ( MQ'7OM)W^SL!' C'<.?UY0W%5[M,AX$]I@+)TJ.ONQN$ M!FBI"2^,XV,;APJ M/7&8YS1@216IZ@ZDACO^P!4CGD>5>P"SVB%.6Z?O6&>O>W@<)@C$(IE0%SPJ M#3B?%^_O\'54ZM4=J5]. M:+43ATRSK][5O;I(EY\7+:S<;;#R3X='X=AD"'VK-Q7XV<]@YPH+G&#MJS[1 M)][/4,E.T:EGO9^9]=.MNYYZ+5^@^/N-JA#!M:+("B\?8G76W02DC99RN8O,$GD#F*EUOJW=]@2F5:!97$?I/,^"GS*N)U;"B5Q2( M0?U;I1.'8M-K-M05*%-SGWV=H#$@M;K5X]HJ0QZ'&7:5];O95V%CD8W%LDP6R3O;']KJM.:STW/4KD,\QB:C* M(_LH'V5=#:S_+6#*U_06WG) ]GNE ;*/5K]3.37;;'S-FJ^5=,=EV.*W+9^]@932,6#(-J$ WV M1DCK!7M&I66#UUMQG+^5YL MASV3&M+4*4-9ZT2A(:T=@V2VS.;:$+5K2:DR]+4G#]CVN_7>VJ7;70$EKT"F']JOJ\FWUV M!')70:M2B5VBS;,T5M@AO?_V\+XLBXV*7O2W-MS@6OUP'WQ5?B-0USG;I4VM75S$A(3RU["_B/=;CNMD@'*0; HSRK[9+#A/RWJ=\([8X M5ADJO1]9[^2I=OQTF[7<-DFT*6<1W?]$K8HVPRH\SY<,7>&LG+!--[ 2.%?H M75UX$'LAS)VT:+:07@:\V9N+;5GU]2JW;1K\\SN'\R/ M2-3G&5/BF%]!&Y=EE!4.[WI8.FHMQ^JU3'VO#_!Z52K7544KX76*6'6J\KCT MEP#KI%/UCD]UJ3I.(TR](ZCCNC]39]K,8%D0DZ^EJ[:^5QV: MU&^KW]Y;J 6!G'2B_LJ.*V73Z=8;K.O6P?;+&(@%X^V\M7?V;LAUU@1K@#TK M!;; ZO0"JT<%F8'5*00>AU]PG]Z$5:EGL6!#U, =.$&BX J\@C$P"^[ 'UB] MB940K.U(L _ "Q#NCZ*].SG@3"*XO)G#4R-R26:9!)G@U]-UPXI=5MS5&6-I M:A'(<_]T1BK-I A_=V63J\+9D@B,PWW'_K(?OWMWBC$,2[V2&0^?G!FULI>P M+A)='Z=R[6Z&S9*9-_&M.^QZ3CEKJ$RHSE#PF>!3"4"YVQ.VA9$U^"I?5HH@ MTZ.!A',*-]_Q2--FY&@GD7S$KQ,2PTYD9XC)&W#ED=).]Y5-@*=R+6^KO)&5 MCNHRXDU(;,WM_A5WG14/4T,]+.GX\&_TPU X$$_/08C?/AL2G*^4;O_FS4$; M<'OQG\VVA,\6KMPW7#;E';V[>6GV6_(Y)+SUV.W)4\;]3@121B4L0>=DMO24 M#7<8!M,4BSF?X?>3G.30_GHW9!F+X9^X-(A5.&=6X0&G3[?PQX.*/VY= L ? M*(_[WP,$C?9XYOU%$YSGVANV6[5'EKP=Q0Q:43TF75R.OI@7_]E!RT+!*]HD M?)9552HX67Q.Z:9%-.+2;FIA?@C>GEP]AKOY^"O? MXWI,CW??5^S'5-DJ V(XR2ZU9=3,FO"3VYKB=#N7'>R\S)UO65O.8EL(B]&1X,2IFFO?CSC!(U,*(+UNF8YYYA0WB+#V([0\' MJ&7O:_P2W?K#??CS ^[/H&@066GWS)' DQ 3O]5\?S5>.3:?,6Z[_636.>)H M[D@=MOWN_*E)8V(5U"CFR9V_?ECI%F5 M_;*&3:O?EZS*9X^(??EJ]36\%A6<2M_%RY_;KW]FJDFN3+3#JTQO>_/S_(/< M>,6666(K2HD=*35\M'?VV@##%P0F=).KT+B7&E/? OM!::K95+HG-/A^Y^*[ MH?I6. M/$X_S\*35R>5'-\#=>]7'7=7 ><4U2_S/75EXG 2Z%T:BR9<:(W+%U0 G\ MS.Q(7BT\M&S2-?OE]@,VBH)^V0*F#23( !7TX-8T@# ;=@+*R("<$!-@ IIV M"V\Z31]5F""G:0:4,6T*X$W' !54$M+TG+;3'*(MH&D1-:AGP-E(TX.Z+L#I M\+(%.H39H!DB2E)+:A82J>V"4[73(>I1$.KP4A(2M:DNU:B:5*OJ1#VH75%^ MW18U8 O4F*.J #I$0^!P,L(X>(7@T *& UJ0 V/("!P/D% 2PG81DQ'A2G-Y;#*,@S&1@(G'[)<^)T2 +G<<>M!;$?%G!0L;]8U7:Z(84? M:9U33!- X[$WRA6_8 WP 37@?%I3(S ^KALW_:\[#RHB01H YWSJ$(#7#Z[0 M<;GJ>>H&=@(F 9B: M4K.Y\"*B/G7,A@DMFS2 :OE4J1_<#%!!9\/!@6HZO064-FG 8[%Z+O#LJ3T2 M9+7K.!N:S04XZJT=&EZ KK8-O1I1_.I@793.]*&.UWV;?;9K! M+6KVMF#3H-I.Y #&C9B.JM6> >.E+K@ &\"Y.?>#\]J_859N#01(. MAJ>U:9O."W'[:+?M5FV[(?>;%MIIVG&;ZKQMM&=VH7[4C1MF%^K[9KL;=Z'. MV];;;.1M[2V^61'-P--K.WV'%_6MM].WVE[?JEHH+6\\3;F7=TX(459[D6Z+ M_=*Y/7>N#MU@.S=LAXP6K.N)V6[=;MIPY^[:+;<#':(NW*KZ%<*$YKV\RXSN MQMUKVW@C:K]]M!,UI#[<39MP4VH'CKG-QNQNU-M[:9MOH82S]_;KAM]Z^WQO MBY9=8T*UYN[?-N!S?VU>[:L).-@("@MN<@LK$VXV##<-* E.6P5Q,,B-!$%U M[P[ABIK#@(1*[1#6M"RZU&_:!F3J2LVH@S;3%MIPVRZH.:M=IUFGUM[:GQIT M4XD +J'8S^F&;))-DKR!,T"(W,#DN19.02U4!/5 %E0 TE(NPH\,F(Q(9-G6 MCUX@)8EA"JB*C! $RHSBP5L* H\/!G>2$?PV'K,91 5;W !M6L(-2!P: HH< M,5"P/4 "D@ 9R @$JI^X@(F1!UB H]@O+/LNS( 7 -D "V_VS5 !WPV'2"T MQ8 :< -WI QA"Q+0!U"0 O 57PYHBZ@@ @2&F$4#X@O.JQBON7 V^(4"J]5- M' AHI0'C5;Y2#G<+;:)_>)7[\='J^&E(#=*!\J3R,2 &[@#N. -OP 40*!$4 M$ 9"03@("0%/- 33@!I4 VM00+$!"2P6L^(3# A$@ I2@2P4 2@ @&$SZWIOD?2$V?!2(,!G"!1G1 M#)4\FE-SD-#$!\PCUPZL@9X_!*' T(? 7; +[SQUH Q8 +N0 J0%W"@G[3S MK"[//<)#4 58G[LD;V3 !) 32)M,*"8]Z\7L'=F35OG.=NB#R2 _E4A)@)B MN.MN0A [&&+CK.2&N[W7Y42$*.X&ZZYYM6]R P<[8Z\A9..QN/9DK M=L*>*-8"4W#K_/HM4'8ST-B9PAUHZS9@L9/V2\(4T$!;U\RJ_2'XIO,!V6^ M"TC4FYVQEX 7<#Y.>P*P 2*A!L3VV1X]V$!MU]HE(;>O]MZ.VI'[<#\?KCT! M8.[8+D:H6!7S% D@OU#WZV'=R8L"2 O0*"S 8+^$Q+ Z(D:/"@!S 'K_H_4 M^ZRI'ZCI@##V"D$;Y$!?\.[@7;R3]S!@WF\$?$\ ]3VHV_5_!-^3!%^_))V] M&DT9_,Y9>$M\/_!^7<$G!OR>E'X$4W!A!GZ^"YPH<0<<^]^0 _A=%]&!!$ N M8CMXOP-D@,3+'^J> %!\ H !":"0Y9OO#N-=0HQ78=\=Q9>;%W_CT M%*(3:BL#7SW"@Z8)OS=22QC1\L,IHUUVJ6814-L8"/-G0<*#^#*O@]#Z43M. MCMV?>'D4\.83@&.7\TH$2IAY.U_/T( "<.S88L_#@:_^Y\<\B(_IQ9W-/Z+> M7C"*^_SA\!Z>*1AY^H3F\7QG__"2_M";=A31VC4"IO_TFOZJH?508WC.? FX M\R5%J>/@Z+0MAKA3Y19DX-0?#%!/Y\]\@Q=.]_T/089S[L^GPS''9_[BF0-M MDFT37H=7<>9>#)J#<^,5"Z4Y,G-P#D&;$Z\;D,V_TB*%,N&\*)VC.?;,S7D_ M3^>HO Z,@5\#(>BY0" (!@$A*(09T!"H^A0P BV WCL$B, 5-+IB, (3H0P@ M =\# E %4 (AJ,)7(25<'Y> E" \W["VA@,$'$64(3A4 )O WT!LR1T$'\ M08\*4R$CK/M[ A.^29GA]QG=5JQTLG #7@ .> &K$B;$ !V 7X9%&Y ,SN9S M9 :[D=39R[#:%G/J431Q%+ 4/D)E"B^L$R0DP M&-))5%@45@8QG(^;P$-6=/]Z7_M%?+,B!>D)4P@538!GR01JI"3;0 OP$[E /6X7@N+8R M0/Q;O\Y'$7;#NO(2V'^^7\5W&9A_EHSN-_,9!0K"&- &Y %K 'H-!P5: &E M M@;"$2!9A@JU 1B" N=\W,0-D!%8 XV^Q_ M6@(-(@),?6>? ,<;B !%H.=A+H@:K 'A5R%,,5:&].&$98=*8@60'\C"S&(]PD944%18"D$$9\@'/,W""FA1LY7(2@%9LS]YS3P M+])@P7 'K#"]'_[7 @B!E0J*$*2L 6G=_$< XG^X7CEH!LQ_!HTX:/<9!:I" M"S 1+'E4RH$0^24M$9^@)R?("R[ &=#_7036PD10%A03Y4+W%^ U%@\=YV(, M*AA%00\(_:D0@?Z""J%]6J!5NA:>?:>0"S 1.%0YV/\R M4P:X01;N@TY %G@&?@U;0!= H5@:1\"2H&8<#!\!"@ *#=IASK MIB%BB,?A?'@@%G$/SA'P(HJ!)>(]&&^@B 7#,_(%%@5_ WW2(M8'D-^CT1YF M-B7?=N@D;@=,HFOH8QXJ]&']N$,P"UD 8L%[G 'O "&03!R)0:)18&6F _Z?C>*IK 2 MNA-BHAM %$P2O(69:!ZZB:PAM[B%PB )A:R MD M0%!WS(6$G&)(^/Q)?P0?9W$R$ =]XF!X%MQWA=_H!\9X!ZJ# P@!2H \8F+0 M!GP$2(&[< :P!3$%I2*#: J>( 2#U[P$: ;D-Q%,&>9?#T('N(H18._G- @9 MB4%1$-!LBM+?'G#6-( /8+*(.X!XG:*T"#\()5\&"&"S7("WH@28)'2*V>+_ ML2T("\Y'7D NKHO>8@WR>KP1X$< 0G!XBY%%7.=ZC B0A 7#+I*+Q=Q TQ5 MB_LB^@#=D+EE0LEHW=0U*@.VHD:D,6 *;A95!/ M*!,%8Z^'FAQX!X$0< 6,C(R-5##E080LH\I(2@2-FH++^#68AFN 87 %/"/4 M"LUX+>(83$T8(894C)^%E$$Y 'EI7A031CB-%\$R4C4H**P>AG(S:G@CW\20 M5-@ N%[GP=>5C6T,$J,B5(Q>XE30)82)E,/S%S=N%AW#1[ .5HW301#($ Z& M=@$*P 0@ 2E 32 #U 0T 4A0$YP-+$"9L1 =U &:'#;A7H+@6U7$I $Y\1" M0 -,CI%0O;<0T'M0QDP@$C NVP8WP>9PCINCB((GV'9/1@Z0'-X,(4%.H I] MA:3CC(()!<:C0.^ICF7"Z+A(_16U8YO'.YX39P+MB">\AE#& M\)@3+(_U'K>WN16/C\A"\!J**)H9?W$[^'B:X^28 VR.;L'C^#HF.)ECS*8] M2HXJBWI0.BZ-U:!A0"4FB2 #)_A)P'IH'9Z XR.THKVZ.79C[1C<_@ZKHZ< MHVWWX. )(0'V:-MI,.JC[9C9$(^B8TEP/(8&9@N4<3RZ!7,*[4C3&(_\P>LH MBVB/OL+]*#L"=W:!KV#;Y8XA 91Q03XBR!SMN,E-CB(D\;A!AI MP/WXX%20 M#\X)R4UDD!0D0_"Q!(]VP6A /!Z//21PESY2&0KD3"@:N(XY 8Z')]![$*3K MZ$ N.--C_CA YC;XXWZ!0!*1&.3F^."L2MKC[3@^;HX2)!6Y1<:.U.,-"=RY M!=_C_YA?T8XXP>C(/W(+^:,)F4"V>>QC@^ ^5HG7BOS8Z1EZQYSY&!I@C@^. MEQ=>') @0!H9&H07FV-.(#KF!'/;](@#%)!EI$W@2#($#PZ?=DZ8+2'D BD[ MVG9#9.DXP6F0D*3,IT)NCIDD+"?TG1,VI"@I.^I[T9_K>#8XC]'?_9@[XHZ1 MI&@P/4J/A^0*$3T*D="?+MDZ3HZI)!19$FR0C.0:.4,R!%"&)7E,:H^I9&4R M.F:/W *.M[X]DZZCB!)>, 1_I.X'#=)[E^-^$$ER"%$D+1E> '>7(W#'+8R3 M461(4$/.D8]('5D&W)'PXR8H9>R1:)TE^4>^AH&D'W/,?3?UGO^82YJ3H@$G MV4N&!!0D<+=.8I$HY ))08:2&>0,^1H6DP_EH^ Z7I&>9"&I0GJ0T9_E^"CP MDM"?0?DHJ).UY!GY4;*0CP+RB$3&DOWD]5A+GI2;S>O8%FAMH@%!V4MBB*(! M0!E,[AJ2.>7T*!.RD2HE5!A%FI+!)/A82.*4O22]MT5JD;VD48DY M4I0S'DQ 1AJ2O23K6$@:D?9D(8E/EI'N)#QI),PM!).C[H@FZ),W MPRUY/YZ39.4,:45"?VVE: Z1I)Q9>=X2+8 S&,AF3K"E9LC'+E7II(M0$CP M1X*1<.5:>53ZE"U -7E8@A/LY VY5?J/)4$'>4F&DK^"(YE(TI4CI&A 2<*5 MD*-FR4*V "(E#4E7A@3#I!,)5\Z5*Z0U259^D,!D8]E5R@'OXU?)-\Z/A5X] M&5X@CR"E(%GR<0OY1;W72TZ2OF7)!T%>EJFD<"D:8)2^PFMH%Z21E:39$$Y: MD(TE],=44I707V;)14:6%N5R:=%%DCCD$3DZ5I?/I$#943*0@F5(R4!>EM ? M)-E2AI @ >>85Q:73.4&>4XZE\,D:1G]>9,KY$&Y/C*-[R1LB4?&C_,D_4@2 MPHU"8D)7)%H-H]_=2.K1)\_?$V ?:">< APP9(01!.3HB-W0COOC LDM:''G MA&0IHDB/C"1>Z4(2E]%E26DFH)=09&C 86:23"0KLED&DRTFR^92FI@@P7X1 M8JJ4F%%KB5".E?>CB))C/H\:IM*G1H8$&^:+Z5KREX:!CNC)@)6?!'@QLB * MW@$J^(RT!.#&-;$I(I47IEV@6VZ6%R9VR66RDM"?4LF*T'N=9%3I7#XXK0CM M" U.CQ>F-4E?LB+O98G98G(8)"5#<#9DF=$?ENE-VI QHQM03\R61P+ 804F9%\",$@>N@?\&(P'-N@)4Q1@ ,(@CT*#O:F*&> MK\!47IB,I"WYX'@VQ"/T)T."!+\C=ZE-RIC1I(^77+*9#$&(J4..F;MC!KEI MZIBE8Y_Y9^J1X(5=(#)&F6,E_W9*GI6>I(@Y3$:40N5U\QKFDF0FG#D]PIBT MXXTS.4J3MJ2(OX9'":U.:[ MF6:&, IG"7DSA)S,9&-)D+1DFDEL1I65I8V#7IZ9ON.,*68. MCS1AB;E5#H\B2KZ)3+(BBF4+:6WNE5ZDBLEI\IFXAY\I;AJ:9YRN0D?>F MEHE6ZG-0QKUY3@J3K,A^84UBG6FFB )M7C?E);798SJ;D65@.3E&G7 EU+EG MFIQ)YZXY3S*=4*:Y.6!^@XAF,W$^S FCG_)7W.F-6<-Z=WQL@HNF7W!EGIVT MY)9Y=LJ4:^:^:5^&F8=GR;EC,I[[YLQYW!!KV"9#X&,VF\4E*WEO-I=(9;SF M4Y9\D>>SR7;*C$MGRMETUI-FIUV0M4F=Q28I"1+@J:I^=8 M65YR;M-CP]ER6I6>)INI4=:>*J9M)T<^GOD>F\E1:AS1)B%9>UJ;9"1@*6:B MC6IFZLF* '=WY?0)?Y:1)^?;:7K&G6=>PCDY@IJ48_!XW?R;5G\8>W$EH M;HJ&9=96$NB6$^7K66I:E,%G"?I2IJ"BY^.900:?R*5%N6^>E"_E[HE*1I>V MII"93%:;)BAV&8+"GO2G\UEPWI\?J*D9;?*2).@1RD%:E&BF>-E0LB*$9#"Y M55*?M*=%.7':=IEDFIAN$H]2Z.O(?F*7)R7U&6J>FF)H$$IZ#J'0)_YI6W*: M"@Z&B5;>EK(C!IIX*BU+:+.9/$*@CZ=S>6MZF=$C-GE?NI!::.89/6J'>:8* MB53^EK>F?@EN"J$H9QKZ@2Z1V.:%&1IDF'TEK1D^4I$+Z ^Y0$Z;<>6AB"0G= M;G 94 YBI=U8=X*):,%\0&CF>\MG(>DZLIX*) 7*82:1KJ9%N7P4H]*D4\5& MGII*Y&=S3MP$YP2.QX$VF41HE'DWSHTF#2JH8$X,\L18Z8J:#1AEPSEK\FO3 MIJM960:?R*;9&7R.GT2FD(E[HI#;PO18?&Z/ALRF6%Z.!"2E \J.AIH"J4!9 M7 J;UB@SVH]NH^/FDTEHVJ)>8$)7-R*($X.S #A\B<)HE$D_PI41CQNIZ>B/ M;^A"\%;Z"N0CE$%MZI]LI>1HDC:C*21;64&NI/UHGMEB=I!Y);6959*DA*5- MZD?"E9@C3&IFHI,A3*"C/=JD[650ZGWRHS^I<[E#0I$])TOJ7-Z5V*A/FGM& M?S-I8NDZ[J!D94IZE49_6>G- .(G,5E>N*19I!8Z:S).W:6'.E8VEDBG>(A M@-F!"@L.*;!9&\(+$>FCD9&:FQPI< <\0I$,9TAJ/JJ6::14BH\"D(#I7'F2 M!H^JY>PHF *,Q:@(>93:I% D#)D?+*;WI3:)DPZE+&E'&J;PI4CI5,IX+J6& MJ5/*@'JEER516I7*!"2E/@>8HJ5IY6KJ>%:4K2E/RII6EJXI5DI(+H^]8UA* MFWJE(B=<:=OMI7.E;ZJ)9KG)V"DH ,:G\5V"#V9B<-/= M!#* ?=I9X*=A(["!"GJG[D(8$-S,%B((1VI8(G-YI6ZY$%RHOX+(>9+* )6H M"+F5ZJ29I8B*F<:DT* LR5G6I"QI\'A7TIF4:2\Y?R)SCZEHBEA^.2=J4JJB M8J@YJF<:0WZ7H"6+BJ("GYQEC8JBX@D )8W:HPZFP2-;":-VIH\(0ZHGT)31 MY_,W.#H+T4*<6!M8"B+ /W@RF $BP*;XB7*6O:.&BG4B<[WC8>$1C,@YJE^JE)R]!1RL D@2JX M,:B*J> D6[E"]J45Y><(I"JF4"JT"$[>E:RJ3EI>(G,C:F9:E5*?C>KI"*MZ MJ#Y>>$BG IH"IJ&I(GH%1,;H]P+^&'%BQ3>L[G]3W520%*R#!)^;X!),$ 1? M@5(7X@%?7:UX&?B)G,(SLEA(&YEB45 AX 'A8_XHHLQU>$ (R47ZFO.=27@Z MX'_!8I>@"%H*$\SE@@)T&=MI_D>L9C+P0L@"+S@-IZ+@^0+D 6>#3*AQ&G,# M:ZB)7[R. $.2FL5:99<'7*(@*L/:KBXC?")BL!+T,/9$"\ :5!LO08[B M%5"&12