************************************************************************ * Myricom GM networking software and documentation * * Copyright (c) 2000, 2001 by Myricom, Inc. * * All rights reserved. See the file `COPYING' for copyright notice. * ************************************************************************ README for GM-1.6.3 GM is a low-level communication layer for Myrinet. ************************************************************************ For updates to this software, visit `http://www.myri.com/scs'. The FAQ file is located at `http://www.myri.com/scs/GM_FAQ.html'. All Myrinet hardware and software technical questions should be directed to help@myri.com. ************************************************************************ Table of Contents: I. Directory structure of GM distribution II. Installation III. GM Ports IV. Thread Safety V. GM Security VI. GM with Different Topologies VII. GM Mapper and Mapper Tools VIII. Recompiling GM MCP (optional) IX. Building User-Level GM programs X. GM Test Programs XI. GM Performance ============================================= I. Directory Structure of the GM distribution ============================================= The basic directory structure for the GM distribution is below: gm-1.6.3 |-- doc GM API Documentation |-- drivers GM Kernel Drivers | |-- aix | |-- freebsd | |-- hpux | |-- irix | |-- linux | |-- macosx | |-- nt4.0 | |-- nt5.0 | |-- solaris2.5 | `-- vxworks |-- examples GM Example Programs |-- include Include directory |-- libgm User-level GM API |-- mcp GM Myrinet Control Program (MCP) | `-- libc |-- mt GM Mapper | |-- ad_hoc_calculator | |-- barrel_simulation | |-- config | |-- deadlock | |-- dijkstra | |-- file_calculator | |-- force | |-- griddle | |-- libmt | |-- mapgen | |-- maps | |-- merlin | |-- mmapper | |-- nway | |-- ping | |-- simple | |-- smapper | |-- switch | |-- tools GM Mapper tools | `-- watch |-- tests GM Test Suite |-- util `-- zlib READMEs are located at various levels in this tree structure. ================ II. Installation ================ GM-1.6.3 is supported on the following operating systems and processors: FreeBSD v4.x for IA32 and Alpha. Linux 2.4 for IA32, IA64, Alpha, and PowerPC Linux 2.2 for IA32, Alpha, and PowerPC An earlier release of GM is supported on the following operating systems and processors: Linux 2.0 for IA32 and Alpha. IRIX 6.5 for Origin 200, Origin 2000, and Octane Mac OS X 10.1.x for PowerPC Solaris 6, 7, 8 for UltraSPARC (32-bit and 64-bit modes). Tru64 (OSF1 V4.0e, V5.0, V5.1, V5.1A) for Alpha. VxWorks for PowerPC Motorola MTX, Force, and CSPI boards. Windows NT 4.0 and Windows 2000 for IA32. Windows XP for IA64. The port of gm-1.6.3 to these operating systems and processors will be available soon. Supported interfaces: LANai7 and LANai9 (If you have LANai4, you will need to upgrade your interface, or use a previous version of GM (gm-1.2.3 for 256K, and gm-1.5.2 for larger memory sizes). For installation instructions of an earlier GM version please refer to the respective README and README- files. Please also note that Linux 2.4 is not supported on gm-1.2.3). Installation instructions for GM 1.6.3 (and operating-system-specific caveats) can be found in the respective README files: * README-FreeBSD * README-linux ============= III. GM Ports ============= In the OS-bypass design, a port is a way for a process to communicate directly with the Myrinet interface through GM. Thus, the number of ports limits the number of processes concurrently using the Myrinet interface. GM ports guarantee fairness and security between GM processes. By default, this software supports 8 ports, and only 5 of these (ports 2,4,5,6,7) are available to non-privileged user-level software. Port 0 is for internal GM use and is never available. Port 1 is used by the GM Mapper when running. Port 3 is used by the IP-over-Myrinet driver, and the IP device must be setup with ifconfig myri0 up To check the status of the GM ports: cd gm-1.5/binary/bin ./gm_board_info If you need more ports than this, please contact help@myri.com. A 16-port configuration can be built for you (depending upon the type of the interface and the amount of embedded memory). Note that these software ports are independent from the hardware ports on the Myrinet switch. ================= IV. Thread Safety ================= GM is not thread-safe but can be used successfully in a multi-threaded application with a few constraints. Basically, each call to GM should be protected by a mutex. As the host overhead of GM is particularly small, the impact on performance of protecting the GM calls is usually negligible. Special provisions need to be made for the blocking receive case. Refer to the FAQ for further details. http://www.myri.com/scs/faq/faq-gm.html#gmq4 ============== V. GM Security ============== By default, the security features of GM are enabled. In particular, opening privileged ports (like the mapper port) is restricted to privileged users. Since GM_ENABLE_SECURITY is set in the driver source code, GM performs checks to ensure that the Unix security model is not violated. For example, only root would be able to open the mapper port and configure the network. With GM_ENABLE_SECURITY set, GM is believed to maintain the integrity of Unix system security. ================================= VI. GM with Different Topologies ================================= * Loopback Configuration * Point-to-Point Configuration * Switch Configuration GM will provide a communication layer in a Myrinet interface in these three topologies: (1) connected to a loopback cable, (2) connected via a cable to another installed interface (point-to-point), or (3) connected via a cable to a switch. When two machines are directly connected by a cable it is NOT possible to send a message from a gm port on one host to itself or to another gm port on the same host. In short, it is not possible for a message leaving the network interface to return to this network interface without a switch in between. Specifically, gm_allsize -v will FAIL to work in this configuration. However, MPICH-over-GM will work in this configuration if the shared memory support is enabled as described in the MPICH-GM documentation, as will IP-over-GM since both of these protocols loop back messages above the GM messaging layer. The loopback and point-to-point configurations require that gm_simpleroute be run before GM applications can be run. Refer to the FAQ for further details. http://www.myri.com/scs/GM_FAQ.html The switch configuration requires the mapper to be run before GM applications can be run. =============================== VII. GM Mapper and Mapper Tools =============================== You will find the GM Mapper source in gm/mt. Details about running the GM Mapper can be found in the respective README- files. For the advanced user, further details about the GM Mapper and its tools can be found in mt/README. To build the mapper tools (simulator etc): [in a source distribution] cd gm/mt make all gm ======================================== VIII. Recompiling the GM MCP (optional) ======================================== The GM distribution contains prebuilt GM Myrinet Control Programs (firmware) for all supported LANai versions. The user doesn't need to compile the firmware. The default prebuilts cover the large majority of configurations. However, you may need to recompile the MCP if you change the number of ports (GM_NUM_PORTS), or if you use unsupported configure options that would force a rebuild. Myricom does not provide support if the firmware has been modified. Building the firmware requires lanai-tools, lanai-gcc and gendat. These tools are available at http://www.myri.com/scs/L3. It is also necessary to configure with --enable-maintainer-mode. =================================== IX. Building User-Level GM Programs =================================== Building GM programs is straightforward. Using the "gm.h" header file and the associated library, one can build GM applications just as with any other library. The only difference is that before the program can be run, the GM driver and any dynamically linked GM library must be installed, and the GM Mapper run. Example programs which call GM can be found in {GM_HOME}/examples/. Architecture-specific details follow. Unix ---- Under Unix, the header file is called "gm.h" and the library is "libgm.a." Unix dynamically linked libraries are not yet supported (with the exception of Solaris and Irix shared libs). Windows 2000 ------------ Under Windows 2000, the header file is "gm.h," the import library is "gm.lib," and the dynamically linked library is "gm.dll." =================== X. GM Test Programs =================== If the GM driver has been installed and the cabling is correct, there will be a green LED lit up on the switch for each connection that is active. If not, check the power supply to the switch, and check that the Myrinet cable is firmly attached at both ends. On the host there should be a green LED and blinking yellow LED on each Myrinet interface. You can then run the five recommended tests, as described in the README.tests file in the binary/bin directory. ================== XI. GM Performance ================== As stated in the document "Guide to Myrinet/PCI Host Interfaces", the Myrinet-2000 interfaces are capable of sustained PCI data rates approaching the limits of the PCI bus (528 MB/s for 64-bit, 66MHz; 264 MB/s for 64-bit, 33HMz or 32-bit, 66MHz; 132 MB/s for 32-bit, 33MHz). However, the data rate to/from system memory will depend upon the host's memory and PCI-bus implementation. The achievable performance can vary greatly from one PCI chipset to another. A host-to-host communication is limited by the weakest component in the data path. If the performance of the PCI bus does not exceed the data rate of the Myrinet link, it will be a bottleneck and thus limit the global data rate of the communication. The maximum performance of GM should be very close to the maximum performance of the PCI bus if the latter is a limitation. From the GM Test Suite, you can run `gm_debug -L' to determine the maximum sustained bandwidth from the PCI bus, and determine if the PCI bus is the bottleneck to your performance. You can also obtain information from the documentation of your motherboard, or run an OS-specific command to determine the type of PCI chipset on the motherboard. For performance comparison, we refer you to the following webpage, which collects performance data of several motherboards and PCI chipsets, http://www.conservativecomputers.com/myrinet/perf.html and encourage you to submit the specifications of your machine (motherboard) with the result of the previous benchmark.