GPGPU-Sim Simulator Version 3.0 (beta) This version of GPGPU-Sim works with CUDA version 3.1 and some earlier versions. It does not (yet) work with CUDA version 4.x. Please see the copyright notice in the file COPYRIGHT distributed with this release in the same directory as this file. If you use this simulator in your research please cite: Ali Bakhoda, George Yuan, Wilson W. L. Fung, Henry Wong, Tor M. Aamodt, Analyzing CUDA Workloads Using a Detailed GPU Simulator, in IEEE International Symposium on Performance Analysis of Systems and Software (ISPASS), Boston, MA, April 19-21, 2009. Please sign up for the google groups page for Q&A (see gpgpu-sim.org), but note that use of this simulator does not imply any level of support. Questions answered on a best effort basis. See Section 2 "INSTALLING, BUILDING and RUNNING GPGPU-Sim" below to get started. 1. CONTRIBUTIONS and HISTORY GPGPU-Sim was created by Tor Aamodt's research group at the University of British Columbia. Many students have contributed including: Wilson W.L. Fung, Ali Bakhoda, George Yuan, Ivan Sham, Henry Wong, Henry Tran, Andrew Turner, Aaron Ariel, Inderpret Singh, Tim Rogers, and others. GPGPU-Sim models the features of a modern graphics processor that are relevant to non-graphics applications. The first version of GPGPU-Sim was used in a MICRO'07 paper and follow-on ACM TACO paper on dynamic warp formation. That version of GPGPU-Sim used the SimpleScalar PISA instruction set for functional simulation, and various configuration files to provide a programming model close to CUDA. Creating benchmarks for the original GPGPU-Sim simulator was a very time consuming process. This motivated the development an interface for directly running CUDA applications to leverage the growing number of applications being developed to use CUDA. We subsequently added support for OpenCL and removed all SimpleScalar code. The interconnection network is simulated using the booksim simulator developed by Bill Dally's research group at Stanford. To produce output that is compatible with the output from running the same CUDA program on the GPU, we have implemented several PTX instructions using the CUDA Math library (part of the CUDA toolkit). Code to interface with the CUDA Math library is contained in cuda-math.h, which also includes several structures derived from vector_types.h (one of the CUDA header files). See file CHANGES for updates in this and earlier versions. 2. INSTALLING, BUILDING and RUNNING GPGPU-Sim GPGPU-Sim was developed on Linux SuSe (this release was tested with SuSe version 11.1) We have tested it also on Ubuntu 10.04.3 LTS (32 bits) and has been used on several other Linux platforms. If you are using Ubuntu, please run the following commands as necessary gpgpu-sim dependencies: "sudo apt-get install build-essentials xutils-dev bison zlib1g-dev flex libboost-all-dev libglu1-mesa-dev" gpgpu-sim documentation: "sudo apt-get install doxygen graphviz" cuda sdk dependencies: "sudo apt-get install libxi-dev libxmu-dev libglut3-dev" Step 1: Ensure you have gcc, g++, make, makedepend, zlib, bison and flex installed on your system. For CUDA 2.x we used gcc version 4.3.2, for CUDA 1.1 we used gcc/g++ version 4.1.3. This version of GPGPU-Sim does not work with CUDA 4.x; We used bison version 2.3, and flex version 2.5.33. Step 2: Download and install the CUDA Toolkit. If you want to run OpenCL on the simulator, download and install NVIDIA's OpenCL driver from . Update your PATH and LD_LIBRARY_PATH as indicated by the NVIDIA install scripts. Note that you will need to use the lib64 directory if you are using a 64bit machine. Step 3: Read the file setup_environment carefully and change it to match your system. Then, from a bash shell, type the following in this directory: source setup_environment Step 4: Type "make" in this directory. This will build the simulator with optimizations enabled so the simulator runs faster. If you want to run the simulator in gdb to debug it, then uncomment the line #export GPGPUSIM_CONFIG=debug in setup_environment and type "source setup_environment" again, then "make" again. Step 4.1 [optional]: Type "make docs" in this directory to build the doxygen documentation. You need to have doxygen and graphviz installed for this to work. "make cleandocs" will remove the generated documentation. The documentation resides at doc/doxygen/html. Step 5: Copy or create the configuration files gpgpusim.conf and \ icnt_config_quadro_islip.txt in your working directory. Sample configuration files are provided under gpgpu-sim/v3.x/configs/*/ Step 6: Build a CUDA appliction (or an OpenCL application). Step 7: Run the application and the device code should now run on the simulator instead of your graphics card. To be able to run the application on your graphics card again, remove $GPGPUSIM_ROOT/lib from your LD_LIBRARY_PATH. Note that for OpenCL applications the NVIDIA driver is required to convert OpenCL ".cl" files to PTX (this in turn may require you have a graphics card, but to run CUDA applications on the simulator a graphics card is not necessary). The resulting PTX can be saved to disk by adding -save_embedded_ptx to your gpgpusim.config file (embedded PTX files with be saved as _0.ptx, _1.ptx, etc...). If you need to run the set of applications in the NVIDIA CUDA SDK code samples then you will need to download, install and build the SDK. 3. USING/MODIFYING THE SIMULATOR Note that doc/GPGPU-Sim_Manual.html has not yet been updated to reflect changes to GPGPU-Sim versus the earlier 2.x versions.