From dac99ef22b4d0e238782731398626fdcf6e5a3a6 Mon Sep 17 00:00:00 2001
From: Tim Rogers <timrogers@gmail.com>
Date: Wed, 20 Feb 2013 22:19:30 -0800
Subject: Merging

//depot/gpgpu_sim_research/fermi_tim/...

to //depot/gpgpu_sim_research/fermi/...

Integrating CLs up to 15295. Descriptions of these CL's are included.

***
A couple changes to aeriel-vision for warp issue plot support
***
More arielvision changes to support the variable-entry length stacked bar chart
***
Properly printing the right resolution of dynamic warp ids
***.
Generalized the scheduler code and added detailed statistics for which warps issue each cycle.
   Verified the execution of the LRR scheduler - still have to get the two level scheduler to work.
***
Implementing the 2lvl scehduler has it has been originally coded.
   LRR on both the inner and outer levels
***
Adding in a debug tracing system to GPGPU-Sim.
   I am sick of writing debug code - then having to comment out, ifdef out or delete it to checkin.
   This also allows for print streams so the user can decided which traces they would like to see.
   Every print in GPGPU-Sim should go through this system - then it will be really easy to only get the information you want and more importantly people will (a) write and (b) checkin code that actually profiles what they are building.
   Reading tracefiles is superiour in many ways to single stepping since you can print the world and just vet the logfile for what you need.
   This also fascilitates advice from the Debugging Rules! book which states that you should never throw away a debugging tool.  Having debug prints that don't get thrown away is big.
***
   Allowing the trace to be specified in the Make.
   Run Make TRACE=0 to compile the code without any traces
***
   Allowing prints from the performance sim to get the actual ptx instruction text
***
Getting the two level scheduler to actaully work...
   What is released in fermi does not work at all - it effectively performs "static warp limit" from my CCWS paper.
   Warps are never demoted from the active list since the functionality checking to see if they are waiting on a longop is completly broken.
   Maybe if the original author had access to the tracing functions this would not have happened.
   The islongop test was completely broken.  It did not mark the register as used, it marked the register number in the instruction as used.
   For example if this instruction was creating a long op:
   ld r6 [r1]
   It would mark register 0 as waiting for a long op (since it is register 0 of the two registers in this instruction), not register 6.
   Additionally, whenever ANY instruction from a warp releases registers, ALL the longops being tracked for this warp get cleared....
   The only way anyone ever thought this worked is if they did not test it....
***
Reworking the warp schedulers to share common code.
   Making the GTX480 use gto by default. I am not sure wht they really use, but it really can't be LRR.
   Also adding in a new file for custom shared trace defines. These are useful when you want a print that has some
   additional criteria or information printed.
   Verified that the schedulers all work to a first order based on traces.
***
Making it so you can run the stats collection scripts from any directory.
   Also allow the caller to specify a stats file instead of just assume its always the same one

[git-p4: depot-paths = "//depot/gpgpu_sim_research/fermi/distribution/": change = 15296]
---
 Makefile                             |   2 +
 aerialvision/guiclasses.py           |  13 +
 aerialvision/lexyacc.py              |   2 +
 aerialvision/organizedata.py         |   9 +-
 configs/GTX480/gpgpusim.config       |  13 +-
 configs/QuadroFX5800/gpgpusim.config |   5 +
 configs/TeslaC2050/gpgpusim.config   |  12 +-
 cuobjdump_to_ptxplus/ptx_parser.h    | 103 +++--
 src/Makefile                         |   5 +
 src/cuda-sim/Makefile                |   5 +
 src/cuda-sim/cuda-sim.cc             |  16 +
 src/cuda-sim/cuda-sim.h              |   1 +
 src/cuda-sim/ptx_ir.cc               |  51 ++-
 src/cuda-sim/ptx_ir.h                |   2 +
 src/cuda-sim/ptx_parser.cc           |  82 ++--
 src/gpgpu-sim/Makefile               |   5 +
 src/gpgpu-sim/gpu-sim.cc             |  25 +-
 src/gpgpu-sim/gpu-sim.h              |   3 +
 src/gpgpu-sim/scoreboard.cc          |  40 +-
 src/gpgpu-sim/scoreboard.h           |   1 +
 src/gpgpu-sim/shader.cc              | 808 ++++++++++++++++++++++-------------
 src/gpgpu-sim/shader.h               | 206 +++++++--
 src/gpgpu-sim/shader_trace.h         |  74 ++++
 src/trace.cc                         |  55 +++
 src/trace.h                          |  79 ++++
 src/trace_streams.tup                |  32 ++
 26 files changed, 1203 insertions(+), 446 deletions(-)
 create mode 100644 src/gpgpu-sim/shader_trace.h
 create mode 100644 src/trace.cc
 create mode 100644 src/trace.h
 create mode 100644 src/trace_streams.tup

diff --git a/Makefile b/Makefile
index aa80c9f..7a9c9bd 100644
--- a/Makefile
+++ b/Makefile
@@ -37,6 +37,8 @@ else
 	export DEBUG=0
 endif
 
+export TRACE?=1
+
 NVCC_PATH=$(shellwhich nvcc)
 ifneq ($(shell which nvcc), "")
 	ifeq ($(DEBUG), 1)
diff --git a/aerialvision/guiclasses.py b/aerialvision/guiclasses.py
index 7e2379b..0a7013b 100644
--- a/aerialvision/guiclasses.py
+++ b/aerialvision/guiclasses.py
@@ -1049,6 +1049,19 @@ class graphManager:
                 Legendname.append('W' + `4*(c-2)+1` +  ':' + `4*(c-1)`)
             BarSequence = range(0,numRows)
 
+        if yAxis == 'WarpIssueSlotBreakdown':
+            Legendname = []
+            for c in range(0, numRows):
+                Legendname.append('W' + `c`)
+            BarSequence = range(0,numRows)
+
+        dynamic_warp_resolution = 32
+        if yAxis == 'WarpIssueDynamicIdBreakdown':
+            Legendname = []
+            for c in range(0, numRows):
+                Legendname.append('W' + `dynamic_warp_resolution*c` + ":" + `dynamic_warp_resolution*(c+1)`)
+            BarSequence = range(0,numRows)
+
         yoff_max = numpy.array([0.0] * numCols)
         for row in range(numRows-1,-1,-1):
             yoff_max += y[row]
diff --git a/aerialvision/lexyacc.py b/aerialvision/lexyacc.py
index c434619..de4e732 100644
--- a/aerialvision/lexyacc.py
+++ b/aerialvision/lexyacc.py
@@ -176,6 +176,8 @@ def parseMe(filename):
         'LDmemlatdist':vc.variable('', 3, 0, 'stackbar'), 
         'STmemlatdist':vc.variable('', 3, 0, 'stackbar'), 
         'WarpDivergenceBreakdown':vc.variable('', 3, 0, 'stackbar'), 
+        'WarpIssueSlotBreakdown':vc.variable('', 3, 0, 'stackbar'), 
+        'WarpIssueDynamicIdBreakdown':vc.variable('', 3, 0, 'stackbar'), 
         'dram_writes_per_cycle':vc.variable('', 1, 0, 'scalar', float),
         'dram_reads_per_cycle' :vc.variable('', 1, 0, 'scalar', float),
         'gpu_stall_by_MSHRwb':vc.variable('', 1, 0, 'scalar'),
diff --git a/aerialvision/organizedata.py b/aerialvision/organizedata.py
index 0b547b3..090b90f 100644
--- a/aerialvision/organizedata.py
+++ b/aerialvision/organizedata.py
@@ -193,12 +193,12 @@ def nullOrganizedShader(nullVar, datatype_c):
     organized = []
     
     #determining how many shader cores are present
-    for x in nullVar:
+    for x in reversed(nullVar):
         if x != 'NULL':
             count += 1
-        else:
-            numPlots = count
+        elif count != 0:
             break
+    numPlots = count
     count = 0
     
     #initializing 2D list
@@ -208,6 +208,9 @@ def nullOrganizedShader(nullVar, datatype_c):
     #filling up list appropriately
     for x in range(0,(len(nullVar))):
         if nullVar[x] == 'NULL':
+            while count < numPlots:
+                organized[count].append(0)
+                count += 1
             count=0
         else:
             organized[count].append(nullVar[x])
diff --git a/configs/GTX480/gpgpusim.config b/configs/GTX480/gpgpusim.config
index 9a295a3..62dd078 100644
--- a/configs/GTX480/gpgpusim.config
+++ b/configs/GTX480/gpgpusim.config
@@ -103,10 +103,12 @@
 
 # Fermi has two schedulers per core
 -gpgpu_num_sched_per_core 2
-# Two Level Scheduler
-#-gpgpu_scheduler tl:16
+# Two Level Scheduler with active and pending pools
+#-gpgpu_scheduler two_level_active:6:0:1
 # Loose round robbin scheduler
--gpgpu_scheduler lrr
+#-gpgpu_scheduler lrr
+# Greedy then oldest scheduler
+-gpgpu_scheduler gto
 
 # stat collection
 -gpgpu_memlatency_stat 14 
@@ -117,3 +119,8 @@
 # power model configs
 -power_simulation_enabled 1
 -gpuwattch_xml_file gpuwattch_gtx480.xml
+
+# tracing functionality
+#-trace_enabled 1
+#-trace_components WARP_SCHEDULER,SCOREBOARD
+#-trace_sampling_core 0
diff --git a/configs/QuadroFX5800/gpgpusim.config b/configs/QuadroFX5800/gpgpusim.config
index 7a2a151..81a5f1f 100644
--- a/configs/QuadroFX5800/gpgpusim.config
+++ b/configs/QuadroFX5800/gpgpusim.config
@@ -72,3 +72,8 @@
 -gpgpu_operand_collector_num_units_sfu 8
 
 -visualizer_enabled 0
+
+# tracing functionality
+#-trace_enabled 1
+#-trace_components WARP_SCHEDULER,SCOREBOARD
+#-trace_sampling_core 0
diff --git a/configs/TeslaC2050/gpgpusim.config b/configs/TeslaC2050/gpgpusim.config
index e161b43..3100cbc 100644
--- a/configs/TeslaC2050/gpgpusim.config
+++ b/configs/TeslaC2050/gpgpusim.config
@@ -106,10 +106,12 @@
 
 # Fermi has two schedulers per core
 -gpgpu_num_sched_per_core 2
-# Two Level Scheduler
-#-gpgpu_scheduler tl:16
+# Two Level Scheduler with active and pending pools
+#-gpgpu_scheduler two_level_active:6:0:1
 # Loose round robbin scheduler
--gpgpu_scheduler lrr
+#-gpgpu_scheduler lrr
+# Greedy then oldest scheduler
+-gpgpu_scheduler gto
 
 # stat collection
 -gpgpu_memlatency_stat 14 
@@ -117,3 +119,7 @@
 -enable_ptx_file_line_stats 1
 -visualizer_enabled 0
 
+# tracing functionality
+#-trace_enabled 1
+#-trace_components WARP_SCHEDULER,SCOREBOARD
+#-trace_sampling_core 0
diff --git a/cuobjdump_to_ptxplus/ptx_parser.h b/cuobjdump_to_ptxplus/ptx_parser.h
index f8b922b..1c96b46 100644
--- a/cuobjdump_to_ptxplus/ptx_parser.h
+++ b/cuobjdump_to_ptxplus/ptx_parser.h
@@ -45,7 +45,7 @@
 #define ARRAY_IDENTIFIER_NO_DIM 2
 #define ARRAY_IDENTIFIER 3
 #define P_DEBUG 0
-#define DPRINTF(...) \
+#define PTX_PARSE_DPRINTF(...) \
    if(P_DEBUG) { \
       printf("(%s:%s:%u) ", __FILE__, __FUNCTION__, __LINE__); \
       printf(__VA_ARGS__); \
@@ -53,7 +53,6 @@
       fflush(stdout); \
    }
 
-
 enum _memory_space_t {
    undefined_space=0,
    reg_space,
@@ -73,43 +72,43 @@ int g_error_detected;
 const char *g_filename = "";
 int g_func_decl;
 
-void set_symtab( void* a ) {DPRINTF(" ");}
-void end_function() {DPRINTF(" ");}
-void add_directive() {DPRINTF(" ");}
-void add_function_arg() {DPRINTF(" ");}
-void add_instruction() {DPRINTF(" ");}
-void add_file( unsigned a, const char *b ) {DPRINTF(" ");}
-void add_variables() {DPRINTF(" ");}
-void set_variable_type() {DPRINTF(" ");}
-void add_option(int a ) {DPRINTF(" ");}
-void add_array_initializer() {DPRINTF(" ");}
-void add_label( const char *a ) {DPRINTF(" ");}
-void set_return() {DPRINTF(" ");}
-void add_opcode( int a ) {DPRINTF(" ");}
-void add_pred( const char *a, int b, int c ) {DPRINTF(" ");}
-void add_scalar_operand( const char *a ) {DPRINTF("%s", a);}
-void add_neg_pred_operand( const char *a ) {DPRINTF(" ");}
-void add_address_operand( const char *a, int b ) {DPRINTF("%s", a);}
-void add_address_operand2( int b ) {DPRINTF(" ");}
-void change_operand_lohi( int a ) {DPRINTF(" ");}
-void change_double_operand_type( int a ) {DPRINTF(" ");}
-void change_operand_neg( ) {DPRINTF(" ");}
-void add_double_operand( const char *a, const char *b ) {DPRINTF(" ");}
-void add_1vector_operand( const char *a ) {DPRINTF(" ");}
-void add_2vector_operand( const char *a, const char *b ) {DPRINTF(" ");}
-void add_3vector_operand( const char *a, const char *b, const char *c ) {DPRINTF(" ");}
-void add_4vector_operand( const char *a, const char *b, const char *c, const char *d ) {DPRINTF(" ");}
-void add_builtin_operand( int a, int b ) {DPRINTF(" ");}
-void add_memory_operand() {DPRINTF(" ");}
-void change_memory_addr_space( const char *a ) {DPRINTF(" ");}
-void add_literal_int( int a ) {DPRINTF(" ");}
-void add_literal_float( float a ) {DPRINTF(" ");}
-void add_literal_double( double a ) {DPRINTF(" ");}
-void add_ptr_spec( enum _memory_space_t spec ) {DPRINTF(" ");}
-void add_extern_spec() {DPRINTF(" ");}
-void add_alignment_spec( int ) {DPRINTF(" ");}
-void add_pragma( const char *a ) {DPRINTF(" ");}
-void add_constptr(const char* identifier1, const char* identifier2, int offset) {DPRINTF(" ");}
+void set_symtab( void* a ) {PTX_PARSE_DPRINTF(" ");}
+void end_function() {PTX_PARSE_DPRINTF(" ");}
+void add_directive() {PTX_PARSE_DPRINTF(" ");}
+void add_function_arg() {PTX_PARSE_DPRINTF(" ");}
+void add_instruction() {PTX_PARSE_DPRINTF(" ");}
+void add_file( unsigned a, const char *b ) {PTX_PARSE_DPRINTF(" ");}
+void add_variables() {PTX_PARSE_DPRINTF(" ");}
+void set_variable_type() {PTX_PARSE_DPRINTF(" ");}
+void add_option(int a ) {PTX_PARSE_DPRINTF(" ");}
+void add_array_initializer() {PTX_PARSE_DPRINTF(" ");}
+void add_label( const char *a ) {PTX_PARSE_DPRINTF(" ");}
+void set_return() {PTX_PARSE_DPRINTF(" ");}
+void add_opcode( int a ) {PTX_PARSE_DPRINTF(" ");}
+void add_pred( const char *a, int b, int c ) {PTX_PARSE_DPRINTF(" ");}
+void add_scalar_operand( const char *a ) {PTX_PARSE_DPRINTF("%s", a);}
+void add_neg_pred_operand( const char *a ) {PTX_PARSE_DPRINTF(" ");}
+void add_address_operand( const char *a, int b ) {PTX_PARSE_DPRINTF("%s", a);}
+void add_address_operand2( int b ) {PTX_PARSE_DPRINTF(" ");}
+void change_operand_lohi( int a ) {PTX_PARSE_DPRINTF(" ");}
+void change_double_operand_type( int a ) {PTX_PARSE_DPRINTF(" ");}
+void change_operand_neg( ) {PTX_PARSE_DPRINTF(" ");}
+void add_double_operand( const char *a, const char *b ) {PTX_PARSE_DPRINTF(" ");}
+void add_1vector_operand( const char *a ) {PTX_PARSE_DPRINTF(" ");}
+void add_2vector_operand( const char *a, const char *b ) {PTX_PARSE_DPRINTF(" ");}
+void add_3vector_operand( const char *a, const char *b, const char *c ) {PTX_PARSE_DPRINTF(" ");}
+void add_4vector_operand( const char *a, const char *b, const char *c, const char *d ) {PTX_PARSE_DPRINTF(" ");}
+void add_builtin_operand( int a, int b ) {PTX_PARSE_DPRINTF(" ");}
+void add_memory_operand() {PTX_PARSE_DPRINTF(" ");}
+void change_memory_addr_space( const char *a ) {PTX_PARSE_DPRINTF(" ");}
+void add_literal_int( int a ) {PTX_PARSE_DPRINTF(" ");}
+void add_literal_float( float a ) {PTX_PARSE_DPRINTF(" ");}
+void add_literal_double( double a ) {PTX_PARSE_DPRINTF(" ");}
+void add_ptr_spec( enum _memory_space_t spec ) {PTX_PARSE_DPRINTF(" ");}
+void add_extern_spec() {PTX_PARSE_DPRINTF(" ");}
+void add_alignment_spec( int ) {PTX_PARSE_DPRINTF(" ");}
+void add_pragma( const char *a ) {PTX_PARSE_DPRINTF(" ");}
+void add_constptr(const char* identifier1, const char* identifier2, int offset) {PTX_PARSE_DPRINTF(" ");}
 
 /*non-dummy stuff below this point*/
 
@@ -127,7 +126,7 @@ bool inTexDirective = false;
 
 
 void add_identifier( const char *a, int b, unsigned c ) {
-	DPRINTF("name=%s", a);
+	PTX_PARSE_DPRINTF("name=%s", a);
 	if(inConstDirective){
 		//g_headerList->getListEnd()
 	}
@@ -135,7 +134,7 @@ void add_identifier( const char *a, int b, unsigned c ) {
 
 void add_function_name( const char *headerInput )
 {
-	DPRINTF("name=%s", headerInput);
+	PTX_PARSE_DPRINTF("name=%s", headerInput);
 	char* headerInfo = (char*) headerInput;
 	std::string compareString = g_headerList->getListEnd().getBase();
 
@@ -149,7 +148,7 @@ void add_function_name( const char *headerInput )
 //void add_space_spec(int headerInput)
 void add_space_spec( enum _memory_space_t spec, int value )
 {
-	DPRINTF("spec=%u", spec);
+	PTX_PARSE_DPRINTF("spec=%u", spec);
 	cuobjdumpInst *instEntry;
 	//static int constmemindex=1;
 	switch(spec)
@@ -187,7 +186,7 @@ void add_space_spec( enum _memory_space_t spec, int value )
 
 void add_scalar_type_spec( int headerInput )
 {
-	DPRINTF(" ");
+	PTX_PARSE_DPRINTF(" ");
 	//const char* compareString = g_headerList->getListEnd().getBase();
 
 	if( (inEntryDirective && inParamDirective) || inTexDirective || inConstDirective)
@@ -252,7 +251,7 @@ void add_scalar_type_spec( int headerInput )
 //void version_header(double versionNumber)
 void add_version_info( float versionNumber, unsigned ext)
 {
-	DPRINTF(" ");
+	PTX_PARSE_DPRINTF(" ");
 	cuobjdumpInst *instEntry = new cuobjdumpInst();
 	instEntry->setBase(".version");
 	g_headerList->add(instEntry);
@@ -269,7 +268,7 @@ void add_version_info( float versionNumber, unsigned ext)
 
 void target_header(char* firstTarget)
 {
-	DPRINTF("%s", firstTarget);
+	PTX_PARSE_DPRINTF("%s", firstTarget);
 	cuobjdumpInst *instEntry = new cuobjdumpInst();
 	instEntry->setBase(".target");
 	g_headerList->add(instEntry);	
@@ -279,7 +278,7 @@ void target_header(char* firstTarget)
 
 void target_header2(char* firstTarget, char* secondTarget)
 {
-	DPRINTF("%s, %s", firstTarget, secondTarget);
+	PTX_PARSE_DPRINTF("%s, %s", firstTarget, secondTarget);
 	cuobjdumpInst *instEntry = new cuobjdumpInst();
 	instEntry->setBase(".target");
 	g_headerList->add(instEntry);	
@@ -291,7 +290,7 @@ void target_header2(char* firstTarget, char* secondTarget)
 
 void target_header3(char* firstTarget, char* secondTarget, char* thirdTarget)
 {
-	DPRINTF("%s, %s, %s", firstTarget, secondTarget, thirdTarget);
+	PTX_PARSE_DPRINTF("%s, %s, %s", firstTarget, secondTarget, thirdTarget);
 	cuobjdumpInst *instEntry = new cuobjdumpInst();
 	instEntry->setBase(".target");
 	g_headerList->add(instEntry);
@@ -305,20 +304,20 @@ void target_header3(char* firstTarget, char* secondTarget, char* thirdTarget)
 
 void start_function( int a )
 {
-	DPRINTF(" ");
+	PTX_PARSE_DPRINTF(" ");
 	inEntryDirective = true;
 }
 
 void* reset_symtab()
 {
-	DPRINTF(" ");
+	PTX_PARSE_DPRINTF(" ");
 	inEntryDirective = false;
 	return (void*) NULL;
 }
 
 void func_header(const char* headerBase)
 {
-	DPRINTF("%s", headerBase);
+	PTX_PARSE_DPRINTF("%s", headerBase);
 	// If start of an entry
 	if((strcmp(headerBase, ".entry")==0)||(strcmp(headerBase, ".func")==0)) {
 		inEntryDirective = true;
@@ -332,7 +331,7 @@ void func_header(const char* headerBase)
 
 void func_header_info(const char* headerInfo)
 {
-	DPRINTF("%s", headerInfo);
+	PTX_PARSE_DPRINTF("%s", headerInfo);
 	//const char* compareString = g_headerList->getListEnd().getBase();
 
 	if(inEntryDirective && !inTexDirective) {
@@ -360,7 +359,7 @@ void func_header_info(const char* headerInfo)
 
 void func_header_info_int(const char* s, int i)
 {
-	DPRINTF("%s %d", s, i);
+	PTX_PARSE_DPRINTF("%s %d", s, i);
 	if(inEntryDirective && !inTexDirective) {
 		g_headerList->getListEnd().addOperand(s);
 		char *buff = (char*) malloc(30*sizeof(char));
diff --git a/src/Makefile b/src/Makefile
index b20186c..cc31764 100644
--- a/src/Makefile
+++ b/src/Makefile
@@ -30,6 +30,7 @@
 # GPGPU-Sim Makefile
 
 DEBUG?=0
+TRACE?=1
 
 include ../version_detection.mk
 
@@ -40,6 +41,10 @@ ifeq ($(GNUC_CPP0X), 1)
     CXXFLAGS += -std=c++0x
 endif
 
+ifeq ($(TRACE),1)
+	CXXFLAGS += -DTRACING_ON=1
+endif
+
 ifneq ($(DEBUG),1)
 	OPTFLAGS += -O3
 else
diff --git a/src/cuda-sim/Makefile b/src/cuda-sim/Makefile
index 3062d99..9e37fe7 100644
--- a/src/cuda-sim/Makefile
+++ b/src/cuda-sim/Makefile
@@ -30,6 +30,7 @@ default: libgpgpu_ptx_sim.a
  
 INTEL=0
 DEBUG?=0
+TRACE?=0
 
 CPP = g++ $(SNOW)
 CC = gcc $(SNOW)
@@ -47,6 +48,10 @@ endif
 OPT += -I$(CUDA_INSTALL_PATH)/include 
 OPT += -fPIC 
 
+ifeq ($(TRACE),1)
+	OPT += -DTRACING_ON=1
+endif
+
 CXX_OPT = $(OPT)
 ifeq ($(INTEL),1)
     CXX_OPT += -std=c++0x
diff --git a/src/cuda-sim/cuda-sim.cc b/src/cuda-sim/cuda-sim.cc
index cf64e82..1dbff50 100644
--- a/src/cuda-sim/cuda-sim.cc
+++ b/src/cuda-sim/cuda-sim.cc
@@ -450,6 +450,22 @@ void ptx_print_insn( address_type pc, FILE *fp )
    assert( finfo );
    finfo->print_insn(pc,fp);
 }
+
+std::string ptx_get_insn_str( address_type pc )
+{
+   std::map<unsigned,function_info*>::iterator f = g_pc_to_finfo.find(pc);
+   if( f == g_pc_to_finfo.end() ) {
+       #define STR_SIZE 255
+       char buff[STR_SIZE];
+       buff[STR_SIZE - 1] = '\0';
+       snprintf(buff, STR_SIZE,"<no instruction at address 0x%x>", pc );
+       return std::string(buff);
+   }
+   function_info *finfo = f->second;
+   assert( finfo );
+   return finfo->get_insn_str(pc);
+}
+
 void ptx_instruction::set_fp_or_int_archop(){
 	op2=UN_OP;
 	if((m_opcode == MEMBAR_OP)||(m_opcode == SSY_OP )||(m_opcode == BRA_OP) || (m_opcode == BAR_OP) || (m_opcode == RET_OP) || (m_opcode == RETP_OP) || (m_opcode == NOP_OP) || (m_opcode == EXIT_OP) || (m_opcode == CALLP_OP) || (m_opcode == CALL_OP)){
diff --git a/src/cuda-sim/cuda-sim.h b/src/cuda-sim/cuda-sim.h
index bcc36d1..261d458 100644
--- a/src/cuda-sim/cuda-sim.h
+++ b/src/cuda-sim/cuda-sim.h
@@ -75,6 +75,7 @@ unsigned ptx_sim_init_thread( kernel_info_t &kernel,
 const warp_inst_t *ptx_fetch_inst( address_type pc );
 const struct gpgpu_ptx_sim_kernel_info* ptx_sim_kernel_info(const class function_info *kernel);
 void ptx_print_insn( address_type pc, FILE *fp );
+std::string ptx_get_insn_str( address_type pc );
 void set_param_gpgpu_num_shaders(int num_shaders);
 
 
diff --git a/src/cuda-sim/ptx_ir.cc b/src/cuda-sim/ptx_ir.cc
index 29f6ff4..0dffe70 100644
--- a/src/cuda-sim/ptx_ir.cc
+++ b/src/cuda-sim/ptx_ir.cc
@@ -35,9 +35,12 @@
 #include <list>
 #include <assert.h>
 #include <algorithm>
+#include "assert.h"
 
 #include "cuda-sim.h"
 
+#define STR_SIZE 1024
+
 unsigned symbol::sm_next_uid = 1;
 
 unsigned symbol::get_uid()
@@ -1188,14 +1191,26 @@ void ptx_instruction::print_insn() const
 
 void ptx_instruction::print_insn( FILE *fp ) const
 {
-   char buf[1024], *p;
-   snprintf(buf,1024,"%s", m_source.c_str());
-   p = strtok(buf,";");
-   if( !is_label() ) 
-      fprintf(fp," PC=0x%03x ", m_PC );
-   else
-      fprintf(fp,"                " );
-   fprintf(fp,"(%s:%u) %s", m_source_file.c_str(), m_source_line, p );
+    fprintf( fp, to_string().c_str() );
+}
+
+std::string ptx_instruction::to_string() const
+{
+   std::string result( m_source );
+   unsigned semi_c_pos = result.find(";");
+   assert( semi_c_pos != std::string::npos );
+   if( !is_label() ) {
+      char buf[ STR_SIZE ];
+      buf[ STR_SIZE - 1 ] = '\0';
+      snprintf( buf, STR_SIZE, " PC=0x%03x ", m_PC );
+      result += std::string( buf );
+   } else
+      result += std::string("                ");
+   char buf[STR_SIZE];
+   buf[STR_SIZE - 1] = '\0';
+   snprintf( buf, STR_SIZE,
+            "(%s:%d) %s", m_source_file.c_str(), m_source_line, m_source.c_str() + semi_c_pos );
+   return result;
 }
 
 unsigned function_info::sm_next_uid = 1;
@@ -1240,6 +1255,26 @@ unsigned function_info::print_insn( unsigned pc, FILE * fp ) const
    return inst_size;
 }
 
+std::string function_info::get_insn_str( unsigned pc ) const
+{
+   unsigned index = pc - m_start_PC;
+   if ( index >= m_instr_mem_size ) {
+      char buff[STR_SIZE];
+      buff[STR_SIZE-1] = '\0';
+      snprintf(buff, STR_SIZE, "<past last instruction (max pc=%u)>", m_start_PC + m_instr_mem_size - 1 );
+      return std::string(buff);
+   } else {
+      if ( m_instr_mem[index] != NULL ) {
+         return m_instr_mem[index]->to_string();
+      } else {
+         char buff[STR_SIZE];
+         buff[STR_SIZE-1] = '\0';
+         snprintf(buff, STR_SIZE, "<no instruction at pc = %u>", pc );
+         return std::string(buff);
+      }
+   }
+}
+
 void gpgpu_ptx_assemble( std::string kname, void *kinfo )
 {
     function_info *func_info = (function_info *)kinfo;
diff --git a/src/cuda-sim/ptx_ir.h b/src/cuda-sim/ptx_ir.h
index 547372d..9a24e47 100644
--- a/src/cuda-sim/ptx_ir.h
+++ b/src/cuda-sim/ptx_ir.h
@@ -786,6 +786,7 @@ public:
 
    void print_insn() const;
    virtual void print_insn( FILE *fp ) const;
+   std::string to_string() const;
    unsigned inst_size() const { return m_inst_size; }
    unsigned uid() const { return m_uid;}
    int get_opcode() const { return m_opcode;}
@@ -1052,6 +1053,7 @@ public:
       return m_name;
    }
    unsigned print_insn( unsigned pc, FILE * fp ) const;
+    std::string get_insn_str( unsigned pc ) const;
    void add_inst( const std::list<ptx_instruction*> &instructions )
    {
       m_instructions = instructions;
diff --git a/src/cuda-sim/ptx_parser.cc b/src/cuda-sim/ptx_parser.cc
index d33007b..a02832d 100644
--- a/src/cuda-sim/ptx_parser.cc
+++ b/src/cuda-sim/ptx_parser.cc
@@ -74,7 +74,7 @@ std::list<operand_info> g_operands;
 std::list<int> g_options;
 std::list<int> g_scalar_type;
 
-#define DPRINTF(...) \
+#define PTX_PARSE_DPRINTF(...) \
    if( g_debug_ir_generation ) { \
       printf(" %s:%u => ",g_filename,ptx_lineno); \
       printf("   (%s:%u) ", __FILE__, __LINE__); \
@@ -126,7 +126,7 @@ symbol_table *init_parser( const char *ptx_filename )
 
 void init_directive_state()
 {
-   DPRINTF("init_directive_state");
+   PTX_PARSE_DPRINTF("init_directive_state");
    g_space_spec=undefined_space;
    g_ptr_spec=undefined_space;
    g_scalar_type_spec=-1;
@@ -141,7 +141,7 @@ void init_directive_state()
 
 void init_instruction_state()
 {
-   DPRINTF("init_instruction_state");
+   PTX_PARSE_DPRINTF("init_instruction_state");
    g_pred = NULL;
    g_neg_pred = 0;
    g_pred_mod = -1;
@@ -156,7 +156,7 @@ static int g_entry_point;
 
 void start_function( int entry_point ) 
 {
-   DPRINTF("start_function");
+   PTX_PARSE_DPRINTF("start_function");
    init_directive_state();
    init_instruction_state();
    g_entry_point = entry_point;
@@ -170,7 +170,7 @@ int g_add_identifier_cached__array_ident;
 
 void add_function_name( const char *name ) 
 {
-   DPRINTF("add_function_name %s %s", name,  ((g_entry_point==1)?"(entrypoint)":((g_entry_point==2)?"(extern)":"")));
+   PTX_PARSE_DPRINTF("add_function_name %s %s", name,  ((g_entry_point==1)?"(entrypoint)":((g_entry_point==2)?"(extern)":"")));
    bool prior_decl = g_global_symbol_table->add_function_decl( name, g_entry_point, &g_func_info, &g_current_symbol_table );
    if( g_add_identifier_cached__identifier ) {
       add_identifier( g_add_identifier_cached__identifier,
@@ -189,7 +189,7 @@ void add_function_name( const char *name )
 
 void add_directive() 
 {
-   DPRINTF("add_directive");
+   PTX_PARSE_DPRINTF("add_directive");
    init_directive_state();
 }
 
@@ -197,7 +197,7 @@ void add_directive()
 
 void end_function() 
 {
-   DPRINTF("end_function");
+   PTX_PARSE_DPRINTF("end_function");
 
    init_directive_state();
    init_instruction_state();
@@ -207,7 +207,7 @@ void end_function()
    gpgpu_ptx_assemble( g_func_info->get_name(), g_func_info );
    g_current_symbol_table = g_global_symbol_table;
 
-   DPRINTF("function %s, PC = %d\n", g_func_info->get_name().c_str(), g_func_info->get_start_PC());
+   PTX_PARSE_DPRINTF("function %s, PC = %d\n", g_func_info->get_name().c_str(), g_func_info->get_start_PC());
 }
 
 #define parse_error(msg, ...) parse_error_impl(__FILE__,__LINE__, msg, ##__VA_ARGS__)
@@ -265,7 +265,7 @@ const ptx_instruction *ptx_instruction_lookup( const char *filename, unsigned li
 
 void add_instruction() 
 {
-   DPRINTF("add_instruction: %s", ((g_opcode>0)?g_opcode_string[g_opcode]:"<label>") );
+   PTX_PARSE_DPRINTF("add_instruction: %s", ((g_opcode>0)?g_opcode_string[g_opcode]:"<label>") );
    assert( g_shader_core_config != 0 );
    ptx_instruction *i = new ptx_instruction( g_opcode, 
                                              g_pred, 
@@ -288,7 +288,7 @@ void add_instruction()
 
 void add_variables() 
 {
-   DPRINTF("add_variables");
+   PTX_PARSE_DPRINTF("add_variables");
    if ( !g_operands.empty() ) {
       assert( g_last_symbol != NULL ); 
       g_last_symbol->add_initializer(g_operands);
@@ -298,7 +298,7 @@ void add_variables()
 
 void set_variable_type()
 {
-   DPRINTF("set_variable_type space_spec=%s scalar_type_spec=%s", 
+   PTX_PARSE_DPRINTF("set_variable_type space_spec=%s scalar_type_spec=%s", 
            g_ptx_token_decode[g_space_spec.get_type()].c_str(), 
            g_ptx_token_decode[g_scalar_type_spec].c_str() );
    parse_assert( g_space_spec != undefined_space, "variable has no space specification" );
@@ -333,7 +333,7 @@ void add_identifier( const char *identifier, int array_dim, unsigned array_ident
       g_add_identifier_cached__array_ident = array_ident;
       return;
    }
-   DPRINTF("add_identifier \"%s\" (%u)", identifier, g_ident_add_uid);
+   PTX_PARSE_DPRINTF("add_identifier \"%s\" (%u)", identifier, g_ident_add_uid);
    g_ident_add_uid++;
    type_info *type = g_var_type;
    type_info_key ti = type->get_key();
@@ -501,27 +501,27 @@ void add_constptr(const char* identifier1, const char* identifier2, int offset)
 void add_function_arg()
 {
    if( g_func_info ) {
-      DPRINTF("add_function_arg \"%s\"", g_last_symbol->name().c_str() );
+      PTX_PARSE_DPRINTF("add_function_arg \"%s\"", g_last_symbol->name().c_str() );
       g_func_info->add_arg(g_last_symbol);
    }
 }
 
 void add_extern_spec() 
 {
-   DPRINTF("add_extern_spec");
+   PTX_PARSE_DPRINTF("add_extern_spec");
    g_extern_spec = 1;
 }
 
 void add_alignment_spec( int spec )
 {
-   DPRINTF("add_alignment_spec");
+   PTX_PARSE_DPRINTF("add_alignment_spec");
    parse_assert( g_alignment_spec == -1, "multiple .align specifiers per variable declaration not allowed." );
    g_alignment_spec = spec;
 }
 
 void add_ptr_spec( enum _memory_space_t spec ) 
 {
-   DPRINTF("add_ptr_spec \"%s\"", g_ptx_token_decode[spec].c_str() );
+   PTX_PARSE_DPRINTF("add_ptr_spec \"%s\"", g_ptx_token_decode[spec].c_str() );
    parse_assert( g_ptr_spec == undefined_space, "multiple ptr space specifiers not allowed." );
    parse_assert( spec == global_space or spec == local_space or spec == shared_space, "invalid space for ptr directive." );
    g_ptr_spec = spec; 
@@ -529,7 +529,7 @@ void add_ptr_spec( enum _memory_space_t spec )
 
 void add_space_spec( enum _memory_space_t spec, int value ) 
 {
-   DPRINTF("add_space_spec \"%s\"", g_ptx_token_decode[spec].c_str() );
+   PTX_PARSE_DPRINTF("add_space_spec \"%s\"", g_ptx_token_decode[spec].c_str() );
    parse_assert( g_space_spec == undefined_space, "multiple space specifiers not allowed." );
    if( spec == param_space_unclassified ) {
       if( g_func_decl ) {
@@ -548,14 +548,14 @@ void add_space_spec( enum _memory_space_t spec, int value )
 
 void add_vector_spec(int spec ) 
 {
-   DPRINTF("add_vector_spec");
+   PTX_PARSE_DPRINTF("add_vector_spec");
    parse_assert( g_vector_spec == -1, "multiple vector specifiers not allowed." );
    g_vector_spec = spec;
 }
 
 void add_scalar_type_spec( int type_spec ) 
 {
-   DPRINTF("add_scalar_type_spec \"%s\"", g_ptx_token_decode[type_spec].c_str());
+   PTX_PARSE_DPRINTF("add_scalar_type_spec \"%s\"", g_ptx_token_decode[type_spec].c_str());
    g_scalar_type.push_back( type_spec );
    if ( g_scalar_type.size() > 1 ) {
       parse_assert( (g_opcode == -1) || (g_opcode == CVT_OP) || (g_opcode == SET_OP) || (g_opcode == SLCT_OP)
@@ -567,7 +567,7 @@ void add_scalar_type_spec( int type_spec )
 
 void add_label( const char *identifier ) 
 {
-   DPRINTF("add_label");
+   PTX_PARSE_DPRINTF("add_label");
    symbol *s = g_current_symbol_table->lookup(identifier);
    if ( s != NULL ) {
       g_label = s;
@@ -583,7 +583,7 @@ void add_opcode( int opcode )
 
 void add_pred( const char *identifier, int neg, int predModifier ) 
 {
-   DPRINTF("add_pred");
+   PTX_PARSE_DPRINTF("add_pred");
    const symbol *s = g_current_symbol_table->lookup(identifier);
    if ( s == NULL ) {
       std::string msg = std::string("predicate \"") + identifier + "\" has no declaration.";
@@ -596,7 +596,7 @@ void add_pred( const char *identifier, int neg, int predModifier )
 
 void add_option( int option ) 
 {
-   DPRINTF("add_option");
+   PTX_PARSE_DPRINTF("add_option");
    g_options.push_back( option );
 }
 
@@ -606,7 +606,7 @@ void add_double_operand( const char *d1, const char *d2 )
    //eg. s[$ofs1+$r0], g[$ofs1+=$r0]
    //TODO: Not sure if I'm going to use this for storing to two destinations or not.
 
-   DPRINTF("add_double_operand");
+   PTX_PARSE_DPRINTF("add_double_operand");
    const symbol *s1 = g_current_symbol_table->lookup(d1);
    const symbol *s2 = g_current_symbol_table->lookup(d2);
    parse_assert( s1 != NULL && s2 != NULL, "component(s) missing declarations.");
@@ -616,7 +616,7 @@ void add_double_operand( const char *d1, const char *d2 )
 void add_1vector_operand( const char *d1 ) 
 {
    // handles the single element vector operand ({%v1}) found in tex.1d instructions
-   DPRINTF("add_1vector_operand");
+   PTX_PARSE_DPRINTF("add_1vector_operand");
    const symbol *s1 = g_current_symbol_table->lookup(d1);
    parse_assert( s1 != NULL, "component(s) missing declarations.");
    g_operands.push_back( operand_info(s1,NULL,NULL,NULL) );
@@ -624,7 +624,7 @@ void add_1vector_operand( const char *d1 )
 
 void add_2vector_operand( const char *d1, const char *d2 ) 
 {
-   DPRINTF("add_2vector_operand");
+   PTX_PARSE_DPRINTF("add_2vector_operand");
    const symbol *s1 = g_current_symbol_table->lookup(d1);
    const symbol *s2 = g_current_symbol_table->lookup(d2);
    parse_assert( s1 != NULL && s2 != NULL, "v2 component(s) missing declarations.");
@@ -633,7 +633,7 @@ void add_2vector_operand( const char *d1, const char *d2 )
 
 void add_3vector_operand( const char *d1, const char *d2, const char *d3 ) 
 {
-   DPRINTF("add_3vector_operand");
+   PTX_PARSE_DPRINTF("add_3vector_operand");
    const symbol *s1 = g_current_symbol_table->lookup(d1);
    const symbol *s2 = g_current_symbol_table->lookup(d2);
    const symbol *s3 = g_current_symbol_table->lookup(d3);
@@ -643,7 +643,7 @@ void add_3vector_operand( const char *d1, const char *d2, const char *d3 )
 
 void add_4vector_operand( const char *d1, const char *d2, const char *d3, const char *d4 ) 
 {
-   DPRINTF("add_4vector_operand");
+   PTX_PARSE_DPRINTF("add_4vector_operand");
    const symbol *s1 = g_current_symbol_table->lookup(d1);
    const symbol *s2 = g_current_symbol_table->lookup(d2);
    const symbol *s3 = g_current_symbol_table->lookup(d3);
@@ -658,13 +658,13 @@ void add_4vector_operand( const char *d1, const char *d2, const char *d3, const
 
 void add_builtin_operand( int builtin, int dim_modifier ) 
 {
-   DPRINTF("add_builtin_operand");
+   PTX_PARSE_DPRINTF("add_builtin_operand");
    g_operands.push_back( operand_info(builtin,dim_modifier) );
 }
 
 void add_memory_operand() 
 {
-   DPRINTF("add_memory_operand");
+   PTX_PARSE_DPRINTF("add_memory_operand");
    assert( !g_operands.empty() );
    g_operands.back().make_memory_operand();
 }
@@ -681,7 +681,7 @@ void change_memory_addr_space(const char *identifier)
 
    bool recognizedType = false;
 
-   DPRINTF("change_memory_addr_space");
+   PTX_PARSE_DPRINTF("change_memory_addr_space");
    assert( !g_operands.empty() );
    if(!strcmp(identifier, "g"))
    {
@@ -724,7 +724,7 @@ void change_operand_lohi( int lohi )
     *2 = hi, reading from highest bits
     */
 
-   DPRINTF("change_operand_lohi");
+   PTX_PARSE_DPRINTF("change_operand_lohi");
    assert( !g_operands.empty() );
 
    g_operands.back().set_operand_lohi(lohi);
@@ -733,7 +733,7 @@ void change_operand_lohi( int lohi )
 
 void set_immediate_operand_type ()
 {
-	 DPRINTF("set_immediate_operand_type");
+	 PTX_PARSE_DPRINTF("set_immediate_operand_type");
 	 assert( !g_operands.empty() );
 	 g_operands.back().set_immediate_addr();
 }
@@ -750,7 +750,7 @@ void change_double_operand_type( int operand_type )
     *3 = reg += immediate
     */
 
-   DPRINTF("change_double_operand_type");
+   PTX_PARSE_DPRINTF("change_double_operand_type");
    assert( !g_operands.empty() );
 
    // For double destination operands, ensure valid instruction
@@ -772,7 +772,7 @@ void change_double_operand_type( int operand_type )
 
 void change_operand_neg( )
 {
-   DPRINTF("change_operand_neg");
+   PTX_PARSE_DPRINTF("change_operand_neg");
    assert( !g_operands.empty() );
 
    g_operands.back().set_operand_neg();
@@ -781,25 +781,25 @@ void change_operand_neg( )
 
 void add_literal_int( int value ) 
 {
-   DPRINTF("add_literal_int");
+   PTX_PARSE_DPRINTF("add_literal_int");
    g_operands.push_back( operand_info(value) );
 }
 
 void add_literal_float( float value ) 
 {
-   DPRINTF("add_literal_float");
+   PTX_PARSE_DPRINTF("add_literal_float");
    g_operands.push_back( operand_info(value) );
 }
 
 void add_literal_double( double value ) 
 {
-   DPRINTF("add_literal_double");
+   PTX_PARSE_DPRINTF("add_literal_double");
    g_operands.push_back( operand_info(value) );
 }
 
 void add_scalar_operand( const char *identifier ) 
 {
-   DPRINTF("add_scalar_operand");
+   PTX_PARSE_DPRINTF("add_scalar_operand");
    const symbol *s = g_current_symbol_table->lookup(identifier);
    if ( s == NULL ) {
       if ( g_opcode == BRA_OP || g_opcode == CALLP_OP) {
@@ -815,7 +815,7 @@ void add_scalar_operand( const char *identifier )
 
 void add_neg_pred_operand( const char *identifier ) 
 {
-   DPRINTF("add_neg_pred_operand");
+   PTX_PARSE_DPRINTF("add_neg_pred_operand");
    const symbol *s = g_current_symbol_table->lookup(identifier);
    if ( s == NULL ) {
        s = g_current_symbol_table->add_variable(identifier,NULL,1,g_filename,ptx_lineno);
@@ -827,7 +827,7 @@ void add_neg_pred_operand( const char *identifier )
 
 void add_address_operand( const char *identifier, int offset ) 
 {
-   DPRINTF("add_address_operand");
+   PTX_PARSE_DPRINTF("add_address_operand");
    const symbol *s = g_current_symbol_table->lookup(identifier);
    if ( s == NULL ) {
       std::string msg = std::string("operand \"") + identifier + "\" has no declaration.";
@@ -838,7 +838,7 @@ void add_address_operand( const char *identifier, int offset )
 
 void add_address_operand2( int offset )
 {
-   DPRINTF("add_address_operand");
+   PTX_PARSE_DPRINTF("add_address_operand");
    g_operands.push_back( operand_info((unsigned)offset) );
 }
 
diff --git a/src/gpgpu-sim/Makefile b/src/gpgpu-sim/Makefile
index 04659aa..b9749e5 100644
--- a/src/gpgpu-sim/Makefile
+++ b/src/gpgpu-sim/Makefile
@@ -29,6 +29,7 @@
 # GPGPU-Sim Makefile
 
 DEBUG?=0
+TRACE?=0
 
 ifeq ($(DEBUG),1)
 	CXXFLAGS = -Wall -DDEBUG
@@ -38,6 +39,10 @@ else
 	CXXFLAGS_L2CACHE = -Wall
 endif
 
+ifeq ($(TRACE),1)
+	CXXFLAGS += -DTRACING_ON=1
+endif
+
 include ../../version_detection.mk
 
 ifeq ($(GNUC_CPP0X), 1)
diff --git a/src/gpgpu-sim/gpu-sim.cc b/src/gpgpu-sim/gpu-sim.cc
index 3fa1d75..3a8cdea 100644
--- a/src/gpgpu-sim/gpu-sim.cc
+++ b/src/gpgpu-sim/gpu-sim.cc
@@ -57,6 +57,7 @@
 #include "../debug.h"
 #include "../gpgpusim_entrypoint.h"
 #include "../cuda-sim/cuda-sim.h"
+#include "../trace.h"
 #include "mem_latency_stat.h"
 #include "power_stat.h"
 #include "visualizer.h"
@@ -260,6 +261,9 @@ void shader_core_config::reg_options(class OptionParser * opp)
     option_parser_register(opp, "-gpgpu_warpdistro_shader", OPT_INT32, &gpgpu_warpdistro_shader, 
                 "Specify which shader core to collect the warp size distribution from", 
                 "-1");
+    option_parser_register(opp, "-gpgpu_warp_issue_shader", OPT_INT32, &gpgpu_warp_issue_shader, 
+                "Specify which shader core to collect the warp issue distribution from", 
+                "0");
     option_parser_register(opp, "-gpgpu_local_mem_map", OPT_BOOL, &gpgpu_local_mem_map, 
                 "Mapping from local memory space address to simulated GPU physical address space (default = enabled)", 
                 "1");
@@ -331,8 +335,11 @@ void shader_core_config::reg_options(class OptionParser * opp)
                             "Number if ldst units (default=1) WARNING: not hooked up to anything",
                              "1");
     option_parser_register(opp, "-gpgpu_scheduler", OPT_CSTR, &gpgpu_scheduler_string,
-                                "Scheduler configuration: lrr|tl:num_active_warps default: lrr",
-                                 "lrr");
+                                "Scheduler configuration: < lrr | gto | two_level_active > "
+                                "If two_level_active:<num_active_warps>:<inner_prioritization>:<outer_prioritization>"
+                                "For complete list of prioritization values see shader.h enum scheduler_prioritization_type"
+                                "Default: gto",
+                                 "gto");
 }
 
 void gpgpu_sim_config::reg_options(option_parser_t opp)
@@ -387,6 +394,17 @@ void gpgpu_sim_config::reg_options(option_parser_t opp)
    option_parser_register(opp, "-visualizer_zlevel", OPT_INT32,
                           &g_visualizer_zlevel, "Compression level of the visualizer output log (0=no comp, 9=highest)",
                           "6");
+    option_parser_register(opp, "-trace_enabled", OPT_BOOL, 
+                          &Trace::enabled, "Turn on traces",
+                          "0");
+    option_parser_register(opp, "-trace_components", OPT_CSTR, 
+                          &Trace::config_str, "comma seperated list of traces to enable. "
+                          "Complete list found in trace_streams.tup. "
+                          "Default none",
+                          "none");
+    option_parser_register(opp, "-trace_sampling_core", OPT_INT32, 
+                          &Trace::sampling_core, "The core which is printed using CORE_DPRINTF. Default 0",
+                          "0");
    ptx_file_line_stats_options(opp);
 }
 
@@ -770,6 +788,7 @@ void gpgpu_sim::gpu_print_stat()
    printf( "gpu_total_sim_rate=%u\n", (unsigned)( ( gpu_tot_sim_insn + gpu_sim_insn ) / elapsed_time ) );
 
    shader_print_l1_miss_stat( stdout );
+   shader_print_scheduler_stat( stdout, true );
 
    m_shader_stats->print(stdout);
 #ifdef GPGPUSIM_POWER_MODEL
@@ -1141,6 +1160,8 @@ void gpgpu_sim::cycle()
                shader_print_runtime_stat( stdout );
             if (m_config.gpu_runtime_stat_flag & GPU_RSTAT_L1MISS) 
                shader_print_l1_miss_stat( stdout );
+            if (m_config.gpu_runtime_stat_flag & GPU_RSTAT_SCHED) 
+               shader_print_scheduler_stat( stdout, false );
          }
       }
 
diff --git a/src/gpgpu-sim/gpu-sim.h b/src/gpgpu-sim/gpu-sim.h
index e121030..8572092 100644
--- a/src/gpgpu-sim/gpu-sim.h
+++ b/src/gpgpu-sim/gpu-sim.h
@@ -30,6 +30,7 @@
 
 #include "../option_parser.h"
 #include "../abstract_hardware_model.h"
+#include "../trace.h"
 #include "addrdec.h"
 #include "shader.h"
 #include <iostream>
@@ -280,6 +281,7 @@ public:
         m_memory_config.init();
         init_clock_domains(); 
         power_config::init();
+        Trace::init();
 
 
         // initialize file name if it is not set 
@@ -416,6 +418,7 @@ private:
    void L2c_print_cache_stat() const;
    void shader_print_runtime_stat( FILE *fout );
    void shader_print_l1_miss_stat( FILE *fout ) const;
+   void shader_print_scheduler_stat( FILE* fout, bool print_dynamic_info ) const;
    void visualizer_printstat();
    void print_shader_cycle_distro( FILE *fout ) const;
 
diff --git a/src/gpgpu-sim/scoreboard.cc b/src/gpgpu-sim/scoreboard.cc
index e2379fd..f412054 100644
--- a/src/gpgpu-sim/scoreboard.cc
+++ b/src/gpgpu-sim/scoreboard.cc
@@ -28,6 +28,7 @@
 #include "scoreboard.h"
 #include "shader.h"
 #include "../cuda-sim/ptx_sim.h"
+#include "shader_trace.h"
 
 
 //Constructor
@@ -60,6 +61,8 @@ void Scoreboard::reserveRegister(unsigned wid, unsigned regnum)
 		printf("Error: trying to reserve an already reserved register (sid=%d, wid=%d, regnum=%d).", m_sid, wid, regnum);
         abort();
 	}
+    SHADER_DPRINTF( SCOREBOARD,
+                    "Reserved Register - warp:%d, reg: %d\n", wid, regnum );
 	reg_table[wid].insert(regnum);
 }
 
@@ -68,6 +71,8 @@ void Scoreboard::releaseRegister(unsigned wid, unsigned regnum)
 {
 	if( !(reg_table[wid].find(regnum) != reg_table[wid].end()) ) 
         return;
+    SHADER_DPRINTF( SCOREBOARD,
+                    "Release register - warp:%d, reg: %d\n", wid, regnum );
 	reg_table[wid].erase(regnum);
 }
 
@@ -77,16 +82,32 @@ const bool Scoreboard::islongop (unsigned warp_id,unsigned regnum) {
 
 void Scoreboard::reserveRegisters(const class warp_inst_t* inst) 
 {
-    for( unsigned r=0; r < 4; r++) 
-        if(inst->out[r] > 0) reserveRegister(inst->warp_id(), inst->out[r]);
+    for( unsigned r=0; r < 4; r++) {
+        if(inst->out[r] > 0) {
+            reserveRegister(inst->warp_id(), inst->out[r]);
+            SHADER_DPRINTF( SCOREBOARD,
+                            "Reserved register - warp:%d, reg: %d\n",
+                            inst->warp_id(),
+                            inst->out[r] );
+        }
+    }
 
     //Keep track of long operations
     if (inst->is_load() &&
     		(	inst->space.get_type() == global_space ||
     			inst->space.get_type() == local_space ||
+                inst->space.get_type() == param_space_kernel ||
+                inst->space.get_type() == param_space_local ||
+                inst->space.get_type() == param_space_unclassified ||
     			inst->space.get_type() == tex_space)){
     	for ( unsigned r=0; r<4; r++) {
-    		if(inst->out[r] > 0) longopregs[inst->warp_id()].insert(r);
+    		if(inst->out[r] > 0) {
+                SHADER_DPRINTF( SCOREBOARD,
+                                "New longopreg marked - warp:%d, reg: %d\n",
+                                inst->warp_id(),
+                                inst->out[r] );
+                longopregs[inst->warp_id()].insert(inst->out[r]);
+            }
     	}
     }
 }
@@ -94,9 +115,16 @@ void Scoreboard::reserveRegisters(const class warp_inst_t* inst)
 // Release registers for an instruction
 void Scoreboard::releaseRegisters(const class warp_inst_t *inst) 
 {
-    for( unsigned r=0; r < 4; r++) 
-        if(inst->out[r] > 0) releaseRegister(inst->warp_id(), inst->out[r]);
-    longopregs[inst->warp_id()].clear();
+    for( unsigned r=0; r < 4; r++) {
+        if(inst->out[r] > 0) {
+            SHADER_DPRINTF( SCOREBOARD,
+                            "Register Released - warp:%d, reg: %d\n",
+                            inst->warp_id(),
+                            inst->out[r] );
+            releaseRegister(inst->warp_id(), inst->out[r]);
+            longopregs[inst->warp_id()].erase(inst->out[r]);
+        }
+    }
 }
 
 /** 
diff --git a/src/gpgpu-sim/scoreboard.h b/src/gpgpu-sim/scoreboard.h
index 8559131..4a76ea3 100644
--- a/src/gpgpu-sim/scoreboard.h
+++ b/src/gpgpu-sim/scoreboard.h
@@ -50,6 +50,7 @@ public:
     const bool islongop(unsigned warp_id, unsigned regnum);
 private:
     void reserveRegister(unsigned wid, unsigned regnum);
+    int get_sid() const { return m_sid; }
 
     unsigned m_sid;
 
diff --git a/src/gpgpu-sim/shader.cc b/src/gpgpu-sim/shader.cc
index 7a822b2..9d974db 100644
--- a/src/gpgpu-sim/shader.cc
+++ b/src/gpgpu-sim/shader.cc
@@ -45,9 +45,11 @@
 #include "icnt_wrapper.h"
 #include <string.h>
 #include <limits.h>
+#include "shader_trace.h"
 
 #define PRIORITIZE_MSHR_OVER_WB 1
 #define MAX(a,b) (((a)>(b))?(a):(b))
+    
 
 /////////////////////////////////////////////////////////////////////////////
 
@@ -72,170 +74,213 @@ shader_core_ctx::shader_core_ctx( class gpgpu_sim *gpu,
    : m_barriers( config->max_warps_per_shader, config->max_cta_per_core ),
      m_dynamic_warp_id(0)
 {
-   m_kernel = NULL;
-   m_gpu = gpu;
-   m_cluster = cluster;
-   m_config = config;
-   m_memory_config = mem_config;
-   m_stats = stats;
-   unsigned warp_size=config->warp_size;
-
-   m_sid = shader_id;
-   m_tpc = tpc_id;
-
-   m_pipeline_reg.reserve(N_PIPELINE_STAGES);
-   for (int j = 0; j<N_PIPELINE_STAGES; j++) {
-      m_pipeline_reg.push_back(register_set(m_config->pipe_widths[j],pipeline_stage_name_decode[j]));
-   }
-
-   m_threadState = (thread_ctx_t*) calloc(sizeof(thread_ctx_t), config->n_thread_per_shader);
-   m_thread = (ptx_thread_info**) calloc(sizeof(ptx_thread_info*), config->n_thread_per_shader);
-
-   m_not_completed = 0;
-   m_active_threads.reset();
-   m_n_active_cta = 0;
-   for (unsigned i = 0; i<MAX_CTA_PER_SHADER; i++  ) 
-      m_cta_status[i]=0;
-   for (unsigned i = 0; i<config->n_thread_per_shader; i++) {
-      m_thread[i]= NULL;
-      m_threadState[i].m_cta_id = -1;
-      m_threadState[i].m_active = false;
-   }
-   
-   // m_icnt = new shader_memory_interface(this,cluster);
+    m_kernel = NULL;
+    m_gpu = gpu;
+    m_cluster = cluster;
+    m_config = config;
+    m_memory_config = mem_config;
+    m_stats = stats;
+    unsigned warp_size=config->warp_size;
+    
+    m_sid = shader_id;
+    m_tpc = tpc_id;
+    
+    m_pipeline_reg.reserve(N_PIPELINE_STAGES);
+    for (int j = 0; j<N_PIPELINE_STAGES; j++) {
+        m_pipeline_reg.push_back(register_set(m_config->pipe_widths[j],pipeline_stage_name_decode[j]));
+    }
+    
+    m_threadState = (thread_ctx_t*) calloc(sizeof(thread_ctx_t), config->n_thread_per_shader);
+    m_thread = (ptx_thread_info**) calloc(sizeof(ptx_thread_info*), config->n_thread_per_shader);
+    
+    m_not_completed = 0;
+    m_active_threads.reset();
+    m_n_active_cta = 0;
+    for ( unsigned i = 0; i<MAX_CTA_PER_SHADER; i++ ) 
+        m_cta_status[i]=0;
+    for (unsigned i = 0; i<config->n_thread_per_shader; i++) {
+        m_thread[i]= NULL;
+        m_threadState[i].m_cta_id = -1;
+        m_threadState[i].m_active = false;
+    }
+    
+    // m_icnt = new shader_memory_interface(this,cluster);
     if ( m_config->gpgpu_perfect_mem ) {
         m_icnt = new perfect_memory_interface(this,cluster);
     } else {
         m_icnt = new shader_memory_interface(this,cluster);
     }
-   m_mem_fetch_allocator = new shader_core_mem_fetch_allocator(shader_id,tpc_id,mem_config);
-
-   // fetch
-   m_last_warp_fetched = 0;
-
-   #define STRSIZE 1024
-   char name[STRSIZE];
-   snprintf(name, STRSIZE, "L1I_%03d", m_sid);
-   m_L1I = new read_only_cache( name,m_config->m_L1I_config,m_sid,get_shader_instruction_cache_id(),m_icnt,IN_L1I_MISS_QUEUE);
-
-   m_warp.resize(m_config->max_warps_per_shader, shd_warp_t(this, warp_size));
-   initilizeSIMTStack(config->max_warps_per_shader,this->get_config()->warp_size);
-   m_scoreboard = new Scoreboard(m_sid, m_config->max_warps_per_shader);
-
-   //scedulers
-   //must currently occur after all inputs have been initialized.
-   std::string sched_config = m_config->gpgpu_scheduler_string;
-   bool tlsched = sched_config.find("tl") != std::string::npos;
-   int tlmaw;
-   if (tlsched){
-	   sscanf(m_config->gpgpu_scheduler_string, "tl:%d", &tlmaw);
-   }
-   for (int i = 0; i < m_config->gpgpu_num_sched_per_core; i++) {
-	   //###CONFIGURATION
-	   if (tlsched){
-		   schedulers.push_back(new TwoLevelScheduler(m_stats,this,m_scoreboard,m_simt_stack,&m_warp,
-				   &m_pipeline_reg[ID_OC_SP],
-				   &m_pipeline_reg[ID_OC_SFU],
-				   &m_pipeline_reg[ID_OC_MEM],
-				   tlmaw));
-	   } else {
-		   schedulers.push_back(new LooseRoundRobbinScheduler(m_stats,this,m_scoreboard,m_simt_stack,&m_warp,
-				   &m_pipeline_reg[ID_OC_SP],
-				   &m_pipeline_reg[ID_OC_SFU],
-				   &m_pipeline_reg[ID_OC_MEM]));
-	   }
-   }
-   for (unsigned i = 0; i < m_warp.size(); i++) {
-       //distribute i's evenly though schedulers;
-       schedulers[i%m_config->gpgpu_num_sched_per_core]->add_supervised_warp_id(i);
-   }
-
-   //op collector configuration
-   enum { SP_CUS, SFU_CUS, MEM_CUS, GEN_CUS };
-   m_operand_collector.add_cu_set(SP_CUS, m_config->gpgpu_operand_collector_num_units_sp, m_config->gpgpu_operand_collector_num_out_ports_sp);
-   m_operand_collector.add_cu_set(SFU_CUS, m_config->gpgpu_operand_collector_num_units_sfu, m_config->gpgpu_operand_collector_num_out_ports_sfu);
-   m_operand_collector.add_cu_set(MEM_CUS, m_config->gpgpu_operand_collector_num_units_mem, m_config->gpgpu_operand_collector_num_out_ports_mem);
-   m_operand_collector.add_cu_set(GEN_CUS, m_config->gpgpu_operand_collector_num_units_gen, m_config->gpgpu_operand_collector_num_out_ports_gen);
-
-   opndcoll_rfu_t::port_vector_t in_ports;
-   opndcoll_rfu_t::port_vector_t out_ports;
-   opndcoll_rfu_t::uint_vector_t cu_sets;
-   for (unsigned i = 0; i < m_config->gpgpu_operand_collector_num_in_ports_sp; i++) {
-       in_ports.push_back(&m_pipeline_reg[ID_OC_SP]);
-       out_ports.push_back(&m_pipeline_reg[OC_EX_SP]);
-       cu_sets.push_back((unsigned)SP_CUS);
-       cu_sets.push_back((unsigned)GEN_CUS);
-       m_operand_collector.add_port(in_ports,out_ports,cu_sets);
-       in_ports.clear(),out_ports.clear(),cu_sets.clear();
-   }
-
-   for (unsigned i = 0; i < m_config->gpgpu_operand_collector_num_in_ports_sfu; i++) {
-       in_ports.push_back(&m_pipeline_reg[ID_OC_SFU]);
-       out_ports.push_back(&m_pipeline_reg[OC_EX_SFU]);
-       cu_sets.push_back((unsigned)SFU_CUS);
-       cu_sets.push_back((unsigned)GEN_CUS);
-       m_operand_collector.add_port(in_ports,out_ports,cu_sets);
-       in_ports.clear(),out_ports.clear(),cu_sets.clear();
-   }
-
-   for (unsigned i = 0; i < m_config->gpgpu_operand_collector_num_in_ports_mem; i++) {
-       in_ports.push_back(&m_pipeline_reg[ID_OC_MEM]);
-       out_ports.push_back(&m_pipeline_reg[OC_EX_MEM]);
-       cu_sets.push_back((unsigned)MEM_CUS);
-       cu_sets.push_back((unsigned)GEN_CUS);                       
-       m_operand_collector.add_port(in_ports,out_ports,cu_sets);
-       in_ports.clear(),out_ports.clear(),cu_sets.clear();
-   }   
-
-
-   for (unsigned i = 0; i < m_config->gpgpu_operand_collector_num_in_ports_gen; i++) {
-       in_ports.push_back(&m_pipeline_reg[ID_OC_SP]);
-       in_ports.push_back(&m_pipeline_reg[ID_OC_SFU]);
-       in_ports.push_back(&m_pipeline_reg[ID_OC_MEM]);
-       out_ports.push_back(&m_pipeline_reg[OC_EX_SP]);
-       out_ports.push_back(&m_pipeline_reg[OC_EX_SFU]);
-       out_ports.push_back(&m_pipeline_reg[OC_EX_MEM]);
-       cu_sets.push_back((unsigned)GEN_CUS);   
-       m_operand_collector.add_port(in_ports,out_ports,cu_sets);
-       in_ports.clear(),out_ports.clear(),cu_sets.clear();
-   }
-
-   m_operand_collector.init( m_config->gpgpu_num_reg_banks, this );
-
-   // execute
-   m_num_function_units = m_config->gpgpu_num_sp_units + m_config->gpgpu_num_sfu_units + 1; // sp_unit, sfu, ldst_unit
-   //m_dispatch_port = new enum pipeline_stage_name_t[ m_num_function_units ];
-   //m_issue_port = new enum pipeline_stage_name_t[ m_num_function_units ];
-
-   //m_fu = new simd_function_unit*[m_num_function_units];
-
-   for (int k = 0; k < m_config->gpgpu_num_sp_units; k++) {
-       m_fu.push_back(new sp_unit( &m_pipeline_reg[EX_WB], m_config, this ));
-       m_dispatch_port.push_back(ID_OC_SP);
-       m_issue_port.push_back(OC_EX_SP);
-   }
-
-   for (int k = 0; k < m_config->gpgpu_num_sfu_units; k++) {
-       m_fu.push_back(new sfu( &m_pipeline_reg[EX_WB], m_config, this ));
-       m_dispatch_port.push_back(ID_OC_SFU);
-       m_issue_port.push_back(OC_EX_SFU);
-   }
-
-   m_ldst_unit = new ldst_unit( m_icnt, m_mem_fetch_allocator, this, &m_operand_collector, m_scoreboard, config, mem_config, stats, shader_id, tpc_id );
-   m_fu.push_back(m_ldst_unit);
-   m_dispatch_port.push_back(ID_OC_MEM);
-   m_issue_port.push_back(OC_EX_MEM);
-
-   assert(m_num_function_units == m_fu.size() and m_fu.size() == m_dispatch_port.size() and m_fu.size() == m_issue_port.size());
-
-   //there are as many result buses as the width of the EX_WB stage
-   num_result_bus = config->pipe_widths[EX_WB];
-   for(unsigned i=0; i<num_result_bus; i++){
-	   this->m_result_bus.push_back(new std::bitset<MAX_ALU_LATENCY>());
-   }
-
-   m_last_inst_gpu_sim_cycle = 0;
-   m_last_inst_gpu_tot_sim_cycle = 0;
+    m_mem_fetch_allocator = new shader_core_mem_fetch_allocator(shader_id,tpc_id,mem_config);
+    
+    // fetch
+    m_last_warp_fetched = 0;
+    
+    #define STRSIZE 1024
+    char name[STRSIZE];
+    snprintf(name, STRSIZE, "L1I_%03d", m_sid);
+    m_L1I = new read_only_cache( name,m_config->m_L1I_config,m_sid,get_shader_instruction_cache_id(),m_icnt,IN_L1I_MISS_QUEUE);
+    
+    m_warp.resize(m_config->max_warps_per_shader, shd_warp_t(this, warp_size));
+    initilizeSIMTStack(config->max_warps_per_shader,this->get_config()->warp_size);
+    m_scoreboard = new Scoreboard(m_sid, m_config->max_warps_per_shader);
+    
+    //scedulers
+    //must currently occur after all inputs have been initialized.
+    std::string sched_config = m_config->gpgpu_scheduler_string;
+    const concrete_scheduler scheduler = sched_config.find("lrr") != std::string::npos ?
+                                         CONCRETE_SCHEDULER_LRR :
+                                         sched_config.find("two_level_active") != std::string::npos ?
+                                         CONCRETE_SCHEDULER_TWO_LEVEL_ACTIVE :
+                                         sched_config.find("gto") != std::string::npos ?
+                                         CONCRETE_SCHEDULER_GTO :
+                                         NUM_CONCRETE_SCHEDULERS;
+    assert ( scheduler != NUM_CONCRETE_SCHEDULERS );
+    
+    for (int i = 0; i < m_config->gpgpu_num_sched_per_core; i++) {
+        switch( scheduler )
+        {
+            case CONCRETE_SCHEDULER_LRR:
+                schedulers.push_back(
+                    new lrr_scheduler( m_stats,
+                                       this,
+                                       m_scoreboard,
+                                       m_simt_stack,
+                                       &m_warp,
+                                       &m_pipeline_reg[ID_OC_SP],
+                                       &m_pipeline_reg[ID_OC_SFU],
+                                       &m_pipeline_reg[ID_OC_MEM],
+                                       i
+                                     )
+                );
+                break;
+            case CONCRETE_SCHEDULER_TWO_LEVEL_ACTIVE:
+                schedulers.push_back(
+                    new two_level_active_scheduler( m_stats,
+                                                    this,
+                                                    m_scoreboard,
+                                                    m_simt_stack,
+                                                    &m_warp,
+                                                    &m_pipeline_reg[ID_OC_SP],
+                                                    &m_pipeline_reg[ID_OC_SFU],
+                                                    &m_pipeline_reg[ID_OC_MEM],
+                                                    i,
+                                                    config->gpgpu_scheduler_string
+                                                  )
+                );
+                break;
+            case CONCRETE_SCHEDULER_GTO:
+                schedulers.push_back(
+                    new gto_scheduler( m_stats,
+                                       this,
+                                       m_scoreboard,
+                                       m_simt_stack,
+                                       &m_warp,
+                                       &m_pipeline_reg[ID_OC_SP],
+                                       &m_pipeline_reg[ID_OC_SFU],
+                                       &m_pipeline_reg[ID_OC_MEM],
+                                       i
+                                     )
+                );
+                break;
+            default:
+                abort();
+        };
+    }
+    
+    for (unsigned i = 0; i < m_warp.size(); i++) {
+        //distribute i's evenly though schedulers;
+        schedulers[i%m_config->gpgpu_num_sched_per_core]->add_supervised_warp_id(i);
+    }
+    for ( int i = 0; i < m_config->gpgpu_num_sched_per_core; ++i ) {
+        schedulers[i]->done_adding_supervised_warps();
+    }
+    
+    //op collector configuration
+    enum { SP_CUS, SFU_CUS, MEM_CUS, GEN_CUS };
+    m_operand_collector.add_cu_set(SP_CUS, m_config->gpgpu_operand_collector_num_units_sp, m_config->gpgpu_operand_collector_num_out_ports_sp);
+    m_operand_collector.add_cu_set(SFU_CUS, m_config->gpgpu_operand_collector_num_units_sfu, m_config->gpgpu_operand_collector_num_out_ports_sfu);
+    m_operand_collector.add_cu_set(MEM_CUS, m_config->gpgpu_operand_collector_num_units_mem, m_config->gpgpu_operand_collector_num_out_ports_mem);
+    m_operand_collector.add_cu_set(GEN_CUS, m_config->gpgpu_operand_collector_num_units_gen, m_config->gpgpu_operand_collector_num_out_ports_gen);
+    
+    opndcoll_rfu_t::port_vector_t in_ports;
+    opndcoll_rfu_t::port_vector_t out_ports;
+    opndcoll_rfu_t::uint_vector_t cu_sets;
+    for (unsigned i = 0; i < m_config->gpgpu_operand_collector_num_in_ports_sp; i++) {
+        in_ports.push_back(&m_pipeline_reg[ID_OC_SP]);
+        out_ports.push_back(&m_pipeline_reg[OC_EX_SP]);
+        cu_sets.push_back((unsigned)SP_CUS);
+        cu_sets.push_back((unsigned)GEN_CUS);
+        m_operand_collector.add_port(in_ports,out_ports,cu_sets);
+        in_ports.clear(),out_ports.clear(),cu_sets.clear();
+    }
+    
+    for (unsigned i = 0; i < m_config->gpgpu_operand_collector_num_in_ports_sfu; i++) {
+        in_ports.push_back(&m_pipeline_reg[ID_OC_SFU]);
+        out_ports.push_back(&m_pipeline_reg[OC_EX_SFU]);
+        cu_sets.push_back((unsigned)SFU_CUS);
+        cu_sets.push_back((unsigned)GEN_CUS);
+        m_operand_collector.add_port(in_ports,out_ports,cu_sets);
+        in_ports.clear(),out_ports.clear(),cu_sets.clear();
+    }
+    
+    for (unsigned i = 0; i < m_config->gpgpu_operand_collector_num_in_ports_mem; i++) {
+        in_ports.push_back(&m_pipeline_reg[ID_OC_MEM]);
+        out_ports.push_back(&m_pipeline_reg[OC_EX_MEM]);
+        cu_sets.push_back((unsigned)MEM_CUS);
+        cu_sets.push_back((unsigned)GEN_CUS);                       
+        m_operand_collector.add_port(in_ports,out_ports,cu_sets);
+        in_ports.clear(),out_ports.clear(),cu_sets.clear();
+    }   
+    
+    
+    for (unsigned i = 0; i < m_config->gpgpu_operand_collector_num_in_ports_gen; i++) {
+        in_ports.push_back(&m_pipeline_reg[ID_OC_SP]);
+        in_ports.push_back(&m_pipeline_reg[ID_OC_SFU]);
+        in_ports.push_back(&m_pipeline_reg[ID_OC_MEM]);
+        out_ports.push_back(&m_pipeline_reg[OC_EX_SP]);
+        out_ports.push_back(&m_pipeline_reg[OC_EX_SFU]);
+        out_ports.push_back(&m_pipeline_reg[OC_EX_MEM]);
+        cu_sets.push_back((unsigned)GEN_CUS);   
+        m_operand_collector.add_port(in_ports,out_ports,cu_sets);
+        in_ports.clear(),out_ports.clear(),cu_sets.clear();
+    }
+    
+    m_operand_collector.init( m_config->gpgpu_num_reg_banks, this );
+    
+    // execute
+    m_num_function_units = m_config->gpgpu_num_sp_units + m_config->gpgpu_num_sfu_units + 1; // sp_unit, sfu, ldst_unit
+    //m_dispatch_port = new enum pipeline_stage_name_t[ m_num_function_units ];
+    //m_issue_port = new enum pipeline_stage_name_t[ m_num_function_units ];
+    
+    //m_fu = new simd_function_unit*[m_num_function_units];
+    
+    for (int k = 0; k < m_config->gpgpu_num_sp_units; k++) {
+        m_fu.push_back(new sp_unit( &m_pipeline_reg[EX_WB], m_config, this ));
+        m_dispatch_port.push_back(ID_OC_SP);
+        m_issue_port.push_back(OC_EX_SP);
+    }
+    
+    for (int k = 0; k < m_config->gpgpu_num_sfu_units; k++) {
+        m_fu.push_back(new sfu( &m_pipeline_reg[EX_WB], m_config, this ));
+        m_dispatch_port.push_back(ID_OC_SFU);
+        m_issue_port.push_back(OC_EX_SFU);
+    }
+    
+    m_ldst_unit = new ldst_unit( m_icnt, m_mem_fetch_allocator, this, &m_operand_collector, m_scoreboard, config, mem_config, stats, shader_id, tpc_id );
+    m_fu.push_back(m_ldst_unit);
+    m_dispatch_port.push_back(ID_OC_MEM);
+    m_issue_port.push_back(OC_EX_MEM);
+    
+    assert(m_num_function_units == m_fu.size() and m_fu.size() == m_dispatch_port.size() and m_fu.size() == m_issue_port.size());
+    
+    //there are as many result buses as the width of the EX_WB stage
+    num_result_bus = config->pipe_widths[EX_WB];
+    for(unsigned i=0; i<num_result_bus; i++){
+        this->m_result_bus.push_back(new std::bitset<MAX_ALU_LATENCY>());
+    }
+    
+    m_last_inst_gpu_sim_cycle = 0;
+    m_last_inst_gpu_tot_sim_cycle = 0;
 }
 
 void shader_core_ctx::reinit(unsigned start_thread, unsigned end_thread, bool reset_not_completed ) 
@@ -433,6 +478,10 @@ void shader_core_stats::event_warp_issued( unsigned s_id, unsigned warp_id, unsi
             m_shader_dynamic_warp_issue_distro[ s_id ].resize(dynamic_warp_id + 1);
         }
         ++m_shader_dynamic_warp_issue_distro[ s_id ][ dynamic_warp_id ];
+        if ( m_shader_warp_slot_issue_distro[ s_id ].size() <= warp_id ) {
+            m_shader_warp_slot_issue_distro[ s_id ].resize(warp_id + 1);
+        }
+        ++m_shader_warp_slot_issue_distro[ s_id ][ warp_id ];
     }
 }
 
@@ -457,6 +506,47 @@ void shader_core_stats::visualizer_print( gzFile visualizer_file )
     }
     gzprintf(visualizer_file,"\n");
 
+    // warp issue breakdown
+    unsigned sid = m_config->gpgpu_warp_issue_shader;
+    gzprintf(visualizer_file, "WarpIssueSlotBreakdown:");
+    unsigned count = 0;
+    unsigned warp_id_issued_sum = 0;
+    for ( std::vector<unsigned>::const_iterator iter = m_shader_warp_slot_issue_distro[ sid ].begin();
+          iter != m_shader_warp_slot_issue_distro[ sid ].end(); iter++, count++ ) {
+        unsigned diff = count < m_last_shader_warp_slot_issue_distro.size() ?
+                        *iter - m_last_shader_warp_slot_issue_distro[ count ] :
+                        *iter;
+        gzprintf( visualizer_file, " %d", diff );
+        warp_id_issued_sum += diff;
+    }
+    m_last_shader_warp_slot_issue_distro = m_shader_warp_slot_issue_distro[ sid ];
+    gzprintf(visualizer_file,"\n");
+
+    #define DYNAMIC_WARP_PRINT_RESOLUTION 32
+    unsigned total_issued_this_resolution = 0;
+    unsigned dynamic_id_issued_sum = 0;
+    count = 0;
+    gzprintf(visualizer_file, "WarpIssueDynamicIdBreakdown:");
+    for ( std::vector<unsigned>::const_iterator iter = m_shader_dynamic_warp_issue_distro[ sid ].begin();
+          iter != m_shader_dynamic_warp_issue_distro[ sid ].end(); iter++, count++ ) {
+        unsigned diff = count < m_last_shader_dynamic_warp_issue_distro.size() ?
+                        *iter - m_last_shader_dynamic_warp_issue_distro[ count ] :
+                        *iter;
+        total_issued_this_resolution += diff;
+        if ( ( count + 1 ) % DYNAMIC_WARP_PRINT_RESOLUTION == 0 ) {
+            gzprintf( visualizer_file, " %d", total_issued_this_resolution );
+            dynamic_id_issued_sum += total_issued_this_resolution;
+            total_issued_this_resolution = 0;
+        }
+    }
+    if ( count % DYNAMIC_WARP_PRINT_RESOLUTION != 0 ) {
+        gzprintf( visualizer_file, " %d", total_issued_this_resolution );
+        dynamic_id_issued_sum += total_issued_this_resolution;
+    }
+    m_last_shader_dynamic_warp_issue_distro = m_shader_dynamic_warp_issue_distro[ sid ];
+    gzprintf(visualizer_file,"\n");
+    assert( warp_id_issued_sum == dynamic_id_issued_sum );
+
     // overall cache miss rates
     gzprintf(visualizer_file, "gpgpu_n_cache_bkconflict: %d\n", gpgpu_n_cache_bkconflict);
     gzprintf(visualizer_file, "gpgpu_n_shmem_bkconflict: %d\n", gpgpu_n_shmem_bkconflict);     
@@ -631,15 +721,113 @@ shd_warp_t& scheduler_unit::warp(int i){
     return (*m_warp)[i];
 }
 
-void LooseRoundRobbinScheduler::cycle()
+
+/**
+ * A general function to order things in a Loose Round Robin way. The simplist use of this
+ * function would be to implement a loose RR scheduler between all the warps assigned to this core.
+ * A more sophisticated usage would be to order a set of "fetch groups" in a RR fashion.
+ * In the first case, the templated class variable would be a simple unsigned int representing the
+ * warp_id.  In the 2lvl case, T could be a struct or a list representing a set of warp_ids.
+ * @param result_list: The resultant list the caller wants returned.  This list is cleared and then populated
+ *                     in a loose round robin way
+ * @param input_list: The list of things that should be put into the result_list. For a simple scheduler
+ *                    this can simply be the m_supervised_warps list.
+ * @param last_issued_from_input:  An iterator pointing the last member in the input_list that issued.
+ *                                 Since this function orders in a RR fashion, the object pointed
+ *                                 to by this iterator will be last in the prioritization list
+ * @param num_warps_to_add: The number of warps you want the scheudler to pick between this cycle.
+ *                          Normally, this will be all the warps availible on the core, i.e.
+ *                          m_supervised_warps.size(). However, a more sophisticated scheduler may wish to
+ *                          limit this number. If the number if < m_supervised_warps.size(), then only
+ *                          the warps with highest RR priority will be placed in the result_list.
+ */
+template < class T >
+void scheduler_unit::order_lrr( std::vector< T >& result_list,
+                                const typename std::vector< T >& input_list,
+                                const typename std::vector< T >::const_iterator& last_issued_from_input,
+                                unsigned num_warps_to_add )
+{
+    assert( num_warps_to_add <= input_list.size() );
+    result_list.clear();
+    typename std::vector< T >::const_iterator iter
+        = ( last_issued_from_input ==  input_list.end() ) ? input_list.begin()
+                                                          : last_issued_from_input + 1;
+
+    for ( unsigned count = 0;
+          count < num_warps_to_add;
+          ++iter, ++count) {
+        if ( iter ==  input_list.end() ) {
+            iter = input_list.begin();
+        }
+        result_list.push_back( *iter );
+    }
+}
+
+/**
+ * A general function to order things in an priority-based way.
+ * The core usage of the function is similar to order_lrr.
+ * The explanation of the additional parameters (beyond order_lrr) explains the further extensions.
+ * @param ordering: An enum that determines how the age function will be treated in prioritization
+ *                  see the definition of OrderingType.
+ * @param priority_function: This function is used to sort the input_list.  It is passed to stl::sort as
+ *                           the sorting fucntion. So, if you wanted to sort a list of integer warp_ids
+ *                           with the oldest warps having the most priority, then the priority_function
+ *                           would compare the age of the two warps.
+ */
+template < class T >
+void scheduler_unit::order_by_priority( std::vector< T >& result_list,
+                                        const typename std::vector< T >& input_list,
+                                        const typename std::vector< T >::const_iterator& last_issued_from_input,
+                                        unsigned num_warps_to_add,
+                                        OrderingType ordering,
+                                        bool (*priority_func)(T lhs, T rhs) )
+{
+    assert( num_warps_to_add <= input_list.size() );
+    result_list.clear();
+    typename std::vector< T > temp = input_list;
+
+
+    if ( ORDERING_GREEDY_THEN_PRIORITY_FUNC == ordering ) {
+        T greedy_value = *last_issued_from_input;
+        result_list.push_back( greedy_value );
+
+        std::sort( temp.begin(), temp.end(), priority_func );
+        typename std::vector< T >::iterator iter = temp.begin();
+        for ( unsigned count = 0; count < num_warps_to_add; ++count, ++iter ) {
+            if ( *iter != greedy_value ) {
+                result_list.push_back( *iter );
+            }
+        }
+    } else if ( ORDERED_PRIORITY_FUNC_ONLY == ordering ) {
+        std::sort( temp.begin(), temp.end(), priority_func );
+        typename std::vector< T >::iterator iter = temp.begin();
+        for ( unsigned count = 0; count < num_warps_to_add; ++count, ++iter ) {
+            result_list.push_back( *iter );
+        }
+    } else {
+        fprintf( stderr, "Unknown ordering - %d\n", ordering );
+        abort();
+    }
+}
+
+void scheduler_unit::cycle()
 {
+    SCHED_DPRINTF( "scheduler_unit::cycle()\n" );
     bool valid_inst = false;  // there was one warp with a valid instruction to issue (didn't require flush due to control hazard)
     bool ready_inst = false;  // of the valid instructions, there was one not waiting for pending register writes
     bool issued_inst = false; // of these we issued one
 
-    for ( unsigned i=0; i < supervised_warps.size(); i++ ) {
-        unsigned supervised_id = (m_last_sup_id_issued+1+i) % supervised_warps.size();
-        unsigned warp_id = supervised_warps[supervised_id];
+    order_warps();
+    for ( std::vector< shd_warp_t* >::const_iterator iter = m_next_cycle_prioritized_warps.begin();
+          iter != m_next_cycle_prioritized_warps.end();
+          iter++ ) {
+        // Don't consider warps that are not yet valid
+        if ( (*iter) == NULL || (*iter)->done_exit() ) {
+            continue;
+        }
+        SCHED_DPRINTF( "Testing (warp_id %u, dynamic_warp_id %u)\n",
+                       (*iter)->get_warp_id(), (*iter)->get_dynamic_warp_id() );
+        unsigned warp_id = (*iter)->get_warp_id();
         unsigned checked=0;
         unsigned issued=0;
         unsigned max_issue = m_shader->m_config->gpgpu_max_insn_issue_per_warp;
@@ -649,15 +837,22 @@ void LooseRoundRobbinScheduler::cycle()
             bool warp_inst_issued = false;
             unsigned pc,rpc;
             m_simt_stack[warp_id]->get_pdom_stack_top_info(&pc,&rpc);
+            SCHED_DPRINTF( "Warp (warp_id %u, dynamic_warp_id %u) has valid instruction (%s)\n",
+                           (*iter)->get_warp_id(), (*iter)->get_dynamic_warp_id(),
+                           ptx_get_insn_str( pc).c_str() );
             if( pI ) {
                 assert(valid);
                 if( pc != pI->pc ) {
+                    SCHED_DPRINTF( "Warp (warp_id %u, dynamic_warp_id %u) control hazard instruction flush\n",
+                                   (*iter)->get_warp_id(), (*iter)->get_dynamic_warp_id() );
                     // control hazard
                     warp(warp_id).set_next_pc(pc);
                     warp(warp_id).ibuffer_flush();
                 } else {
                     valid_inst = true;
                     if ( !m_scoreboard->checkCollision(warp_id, pI) ) {
+                        SCHED_DPRINTF( "Warp (warp_id %u, dynamic_warp_id %u) passes scoreboard\n",
+                                       (*iter)->get_warp_id(), (*iter)->get_dynamic_warp_id() );
                         ready_inst = true;
                         const active_mask_t &active_mask = m_simt_stack[warp_id]->get_active_mask();
                         assert( warp(warp_id).inst_in_pipeline() );
@@ -686,19 +881,40 @@ void LooseRoundRobbinScheduler::cycle()
                                 }
                             } 
                         }
+                    } else {
+                        SCHED_DPRINTF( "Warp (warp_id %u, dynamic_warp_id %u) fails scoreboard\n",
+                                       (*iter)->get_warp_id(), (*iter)->get_dynamic_warp_id() );
                     }
                 }
             } else if( valid ) {
                // this case can happen after a return instruction in diverged warp
+               SCHED_DPRINTF( "Warp (warp_id %u, dynamic_warp_id %u) return from diverged warp flush\n",
+                              (*iter)->get_warp_id(), (*iter)->get_dynamic_warp_id() );
                warp(warp_id).set_next_pc(pc);
                warp(warp_id).ibuffer_flush();
             }
-            if(warp_inst_issued)
-               warp(warp_id).ibuffer_step();
+            if(warp_inst_issued) {
+                SCHED_DPRINTF( "Warp (warp_id %u, dynamic_warp_id %u) issued %u instructions\n",
+                               (*iter)->get_warp_id(),
+                               (*iter)->get_dynamic_warp_id(),
+                               issued );
+                do_on_warp_issued( warp_id, issued, iter );
+            }
             checked++;
         }
         if ( issued ) {
-            m_last_sup_id_issued=supervised_id;
+            // This might be a bit inefficient, but we need to maintain
+            // two ordered list for proper scheduler execution.
+            // We could remove the need for this loop by associating a
+            // supervised_is index with each entry in the m_next_cycle_prioritized_warps
+            // vector. For now, just run through until you find the right warp_id
+            for ( std::vector< shd_warp_t* >::const_iterator supervised_iter = m_supervised_warps.begin();
+                  supervised_iter != m_supervised_warps.end();
+                  ++supervised_iter ) {
+                if ( *iter == *supervised_iter ) {
+                    m_last_supervised_issued = supervised_iter;
+                }
+            }
             break;
         } 
     }
@@ -712,138 +928,110 @@ void LooseRoundRobbinScheduler::cycle()
         m_stats->shader_cycle_distro[2]++; // pipeline stalled
 }
 
-void TwoLevelScheduler::cycle() {
-	//Move waiting warps to pendingWarps
-	for (	std::list<int>::iterator iter = activeWarps.begin();
-			iter != activeWarps.end();
-			iter ++) {
-		bool waiting = warp(*iter).waiting();
+void scheduler_unit::do_on_warp_issued( unsigned warp_id,
+                                        unsigned num_issued,
+                                        const std::vector< shd_warp_t* >::const_iterator& prioritized_iter )
+{
+    m_stats->event_warp_issued( m_shader->get_sid(),
+                                warp_id,
+                                num_issued,
+                                warp(warp_id).get_dynamic_warp_id() );
+    warp(warp_id).ibuffer_step();
+}
+
+bool scheduler_unit::sort_warps_by_oldest_dynamic_id(shd_warp_t* lhs, shd_warp_t* rhs)
+{
+    if (rhs && lhs) {
+        return lhs->get_dynamic_warp_id() < rhs->get_dynamic_warp_id();
+    } else {
+        return lhs < rhs;
+    }
+}
+
+void lrr_scheduler::order_warps()
+{
+    order_lrr( m_next_cycle_prioritized_warps,
+               m_supervised_warps,
+               m_last_supervised_issued,
+               m_supervised_warps.size() );
+}
+
+void gto_scheduler::order_warps()
+{
+    order_by_priority( m_next_cycle_prioritized_warps,
+                       m_supervised_warps,
+                       m_last_supervised_issued,
+                       m_supervised_warps.size(),
+                       ORDERING_GREEDY_THEN_PRIORITY_FUNC,
+                       scheduler_unit::sort_warps_by_oldest_dynamic_id );
+}
+
+void
+two_level_active_scheduler::do_on_warp_issued( unsigned warp_id,
+                                               unsigned num_issued,
+                                               const std::vector< shd_warp_t* >::const_iterator& prioritized_iter )
+{
+    scheduler_unit::do_on_warp_issued( warp_id, num_issued, prioritized_iter );
+    if ( SCHEDULER_PRIORITIZATION_LRR == m_inner_level_prioritization ) {
+        std::vector< shd_warp_t* > new_active; 
+        order_lrr( new_active,
+                   m_next_cycle_prioritized_warps,
+                   prioritized_iter,
+                   m_next_cycle_prioritized_warps.size() );
+        m_next_cycle_prioritized_warps = new_active;
+    } else {
+        fprintf( stderr,
+                 "Unimplemented m_inner_level_prioritization: %d\n",
+                 m_inner_level_prioritization );
+        abort();
+    }
+}
+
+void two_level_active_scheduler::order_warps()
+{
+	//Move waiting warps to m_pending_warps
+    unsigned num_demoted = 0;
+	for (	std::vector< shd_warp_t* >::iterator iter = m_next_cycle_prioritized_warps.begin();
+			iter != m_next_cycle_prioritized_warps.end(); ) {
+		bool waiting = (*iter)->waiting();
 		for (int i=0; i<4; i++){
-			const warp_inst_t* inst = warp(*iter).ibuffer_next_inst();
+			const warp_inst_t* inst = (*iter)->ibuffer_next_inst();
 			//Is the instruction waiting on a long operation?
-			if ( inst && inst->in[i] > 0 && this->m_scoreboard->islongop(*iter, inst->in[i])){
+			if ( inst && inst->in[i] > 0 && this->m_scoreboard->islongop((*iter)->get_warp_id(), inst->in[i])){
 				waiting = true;
 			}
 		}
 
-		if(waiting){
-			pendingWarps.push_back(*iter);
-			activeWarps.erase(iter);
-			break;
-		}
-	}
-
-	//If there is space in activeWarps, try to find ready warps in pendingWarps
-	if (this->activeWarps.size() < maxActiveWarps){
-		for (	std::list<int>::iterator iter = pendingWarps.begin();
-				iter != pendingWarps.end();
-				iter++){
-			if(!warp(*iter).waiting()){
-				activeWarps.push_back(*iter);
-				pendingWarps.erase(iter);
-				break;
-			}
-		}
-	}
-
-	//Do the scheduling only from activeWarps
-	//If you schedule an instruction, move it to the end of the list
-
-	bool valid_inst = false;  // there was one warp with a valid instruction to issue (didn't require flush due to control hazard)
-	bool ready_inst = false;  // of the valid instructions, there was one not waiting for pending register writes
-	bool issued_inst = false; // of these we issued one
-
-	for (	std::list<int>::iterator warp_id = activeWarps.begin();
-			warp_id != activeWarps.end();
-			warp_id++) {
-		unsigned checked=0;
-		unsigned issued=0;
-		unsigned max_issue = m_shader->m_config->gpgpu_max_insn_issue_per_warp;
-		while(!warp(*warp_id).waiting() && !warp(*warp_id).ibuffer_empty() && (checked < max_issue) && (checked <= issued) && (issued < max_issue) ) {
-			const warp_inst_t *pI = warp(*warp_id).ibuffer_next_inst();
-			bool valid = warp(*warp_id).ibuffer_next_valid();
-			bool warp_inst_issued = false;
-			unsigned pc,rpc;
-			m_simt_stack[*warp_id]->get_pdom_stack_top_info(&pc,&rpc);
-			if( pI ) {
-				assert(valid);
-				if( pc != pI->pc ) {
-					// control hazard
-					warp(*warp_id).set_next_pc(pc);
-					warp(*warp_id).ibuffer_flush();
-				} else {
-					valid_inst = true;
-					if ( !m_scoreboard->checkCollision(*warp_id, pI) ) {
-						ready_inst = true;
-						const active_mask_t &active_mask = m_simt_stack[*warp_id]->get_active_mask();
-						assert( warp(*warp_id).inst_in_pipeline() );
-						if ( (pI->op == LOAD_OP) || (pI->op == STORE_OP) || (pI->op == MEMORY_BARRIER_OP) ) {
-							if( m_mem_out->has_free() ) {
-								m_shader->issue_warp(*m_mem_out,pI,active_mask,*warp_id);
-								issued++;
-								issued_inst=true;
-								warp_inst_issued = true;
-								// Move it to pendingWarps
-								unsigned currwarp = *warp_id;
-								activeWarps.erase(warp_id);
-								activeWarps.push_back(currwarp);
-							}
-						} else {
-							bool sp_pipe_avail = m_sp_out->has_free();
-							bool sfu_pipe_avail = m_sfu_out->has_free();
-							if( sp_pipe_avail && (pI->op != SFU_OP) ) {
-								// always prefer SP pipe for operations that can use both SP and SFU pipelines
-								m_shader->issue_warp(*m_sp_out,pI,active_mask,*warp_id);
-								issued++;
-								issued_inst=true;
-								warp_inst_issued = true;
-								//Move it to end of the activeWarps
-								unsigned currwarp = *warp_id;
-								activeWarps.erase(warp_id);
-								activeWarps.push_back(currwarp);
-							} else if ( (pI->op == SFU_OP) || (pI->op == ALU_SFU_OP) ) {
-								if( sfu_pipe_avail ) {
-									m_shader->issue_warp(*m_sfu_out,pI,active_mask,*warp_id);
-									issued++;
-									issued_inst=true;
-									warp_inst_issued = true;
-									//Move it to end of the activeWarps
-									unsigned currwarp = *warp_id;
-									activeWarps.erase(warp_id);
-									activeWarps.push_back(currwarp);
-
-								}
-							}
-						}
-					}
-				}
-			} else if( valid ) {
-				// this case can happen after a return instruction in diverged warp
-				warp(*warp_id).set_next_pc(pc);
-				warp(*warp_id).ibuffer_flush();
-			}
-			if(warp_inst_issued)
-				warp(*warp_id).ibuffer_step();
-			checked++;
-		}
-		if ( issued ) {
-			break;
-		}
+		if( waiting ) {
+			m_pending_warps.push_back(*iter);
+			iter = m_next_cycle_prioritized_warps.erase(iter);
+            SCHED_DPRINTF( "DEMOTED warp_id=%d, dynamic_warp_id=%d\n",
+                           (*iter)->get_warp_id(),
+                           (*iter)->get_dynamic_warp_id() );
+            ++num_demoted;
+		} else {
+            ++iter;
+        }
 	}
 
-    // tgrogers - fixing a warning about unused variables at the top of this fucntion...
-    // This scheduler has A LOT of copied code from the LRR scheduler.
-    // TODO - this thing really needs to be re-written in a modular way.
-    // For now, to get rid of the warnings I am including the stats epilogue
-    // from the original scheduler function that the author of this function forgot to copy/paste
-    //
-    // issue stall statistics:
-    if( !valid_inst ) 
-        m_stats->shader_cycle_distro[0]++; // idle or control hazard
-    else if( !ready_inst ) 
-        m_stats->shader_cycle_distro[1]++; // waiting for RAW hazards (possibly due to memory) 
-    else if( !issued_inst ) 
-        m_stats->shader_cycle_distro[2]++; // pipeline stalled
+	//If there is space in m_next_cycle_prioritized_warps, promote the next m_pending_warps
+    unsigned num_promoted = 0;
+    if ( SCHEDULER_PRIORITIZATION_SRR == m_outer_level_prioritization ) {
+        while ( m_next_cycle_prioritized_warps.size() < m_max_active_warps ) {
+            m_next_cycle_prioritized_warps.push_back(m_pending_warps.front());
+            m_pending_warps.pop_front();
+            SCHED_DPRINTF( "PROMOTED warp_id=%d, dynamic_warp_id=%d\n",
+                           (m_next_cycle_prioritized_warps.back())->get_warp_id(),
+                           (m_next_cycle_prioritized_warps.back())->get_dynamic_warp_id() );
+            ++num_promoted;
+    	}
+    } else {
+        fprintf( stderr,
+                 "Unimplemented m_outer_level_prioritization: %d\n",
+                 m_outer_level_prioritization );
+        abort();
+    }
+    assert( num_promoted == num_demoted );
 }
 
 void shader_core_ctx::read_operands()
@@ -1727,6 +1915,46 @@ void gpgpu_sim::shader_print_runtime_stat( FILE *fout )
 }
 
 
+void gpgpu_sim::shader_print_scheduler_stat( FILE* fout, bool print_dynamic_info ) const
+{
+    // Print out the stats from the sampling shader core
+    const unsigned scheduler_sampling_core = m_shader_config->gpgpu_warp_issue_shader;
+    #define STR_SIZE 55
+    char name_buff[ STR_SIZE ];
+    name_buff[ STR_SIZE - 1 ] = '\0';
+    const std::vector< unsigned >& distro
+        = print_dynamic_info ?
+          m_shader_stats->get_dynamic_warp_issue()[ scheduler_sampling_core ] :
+          m_shader_stats->get_warp_slot_issue()[ scheduler_sampling_core ];
+    if ( print_dynamic_info ) {
+        snprintf( name_buff, STR_SIZE - 1, "dynamic_warp_id" );
+    } else {
+        snprintf( name_buff, STR_SIZE - 1, "warp_id" );
+    }
+    fprintf( fout,
+             "Shader %d %s issue ditsribution:\n",
+             scheduler_sampling_core,
+             name_buff );
+    const unsigned num_warp_ids = distro.size();
+    // First print out the warp ids
+    fprintf( fout, "%s:\n", name_buff );
+    for ( unsigned warp_id = 0;
+          warp_id < num_warp_ids;
+          ++warp_id  ) {
+        fprintf( fout, "%d, ", warp_id );
+    }
+
+    fprintf( fout, "\ndistro:\n" );
+    // Then print out the distribution of instuctions issued
+    for ( std::vector< unsigned >::const_iterator iter = distro.begin();
+          iter != distro.end();
+          iter++ ) {
+        fprintf( fout, "%d, ", *iter );
+    }
+    fprintf( fout, "\n" );
+}
+
+
 void gpgpu_sim::shader_print_l1_miss_stat( FILE *fout ) const
 {
    unsigned total_d1_misses = 0, total_d1_accesses = 0;
diff --git a/src/gpgpu-sim/shader.h b/src/gpgpu-sim/shader.h
index c74fa96..952651e 100644
--- a/src/gpgpu-sim/shader.h
+++ b/src/gpgpu-sim/shader.h
@@ -259,6 +259,27 @@ class shader_core_ctx;
 class shader_core_config;
 class shader_core_stats;
 
+enum scheduler_prioritization_type
+{
+    SCHEDULER_PRIORITIZATION_LRR = 0, // Loose Round Robin
+    SCHEDULER_PRIORITIZATION_SRR, // Strict Round Robin
+    SCHEDULER_PRIORITIZATION_GTO, // Greedy Then Oldest
+    SCHEDULER_PRIORITIZATION_GTLRR, // Greedy Then Loose Round Robin
+    SCHEDULER_PRIORITIZATION_GTY, // Greedy Then Youngest
+    SCHEDULER_PRIORITIZATION_OLDEST, // Oldest First
+    SCHEDULER_PRIORITIZATION_YOUNGEST, // Youngest First
+};
+
+// Each of these corresponds to a string value in the gpgpsim.config file
+// For example - to specify the LRR scheudler the config must contain lrr
+enum concrete_scheduler
+{
+    CONCRETE_SCHEDULER_LRR = 0,
+    CONCRETE_SCHEDULER_GTO,
+    CONCRETE_SCHEDULER_TWO_LEVEL_ACTIVE,
+    NUM_CONCRETE_SCHEDULERS
+};
+
 class scheduler_unit { //this can be copied freely, so can be used in std containers.
 public:
     scheduler_unit(shader_core_stats* stats, shader_core_ctx* shader, 
@@ -266,21 +287,74 @@ public:
                    std::vector<shd_warp_t>* warp, 
                    register_set* sp_out,
                    register_set* sfu_out,
-                   register_set* mem_out) 
-        : supervised_warps(), m_last_sup_id_issued(0), m_stats(stats), m_shader(shader),
+                   register_set* mem_out,
+                   int id) 
+        : m_supervised_warps(), m_stats(stats), m_shader(shader),
         m_scoreboard(scoreboard), m_simt_stack(simt), /*m_pipeline_reg(pipe_regs),*/ m_warp(warp),
-        m_sp_out(sp_out),m_sfu_out(sfu_out),m_mem_out(mem_out){}
+        m_sp_out(sp_out),m_sfu_out(sfu_out),m_mem_out(mem_out), m_id(id){}
     virtual ~scheduler_unit(){}
     virtual void add_supervised_warp_id(int i) {
-        supervised_warps.push_back(i);
+        m_supervised_warps.push_back(&warp(i));
+    }
+    virtual void done_adding_supervised_warps() {
+        m_last_supervised_issued = m_supervised_warps.end();
     }
 
-    virtual void cycle()=0;
+
+    // The core scheduler cycle method is meant to be common between
+    // all the derived schedulers.  The scheduler's behaviour can be
+    // modified by changing the contents of the m_next_cycle_prioritized_warps list.
+    void cycle();
+
+    // These are some common ordering fucntions that the
+    // higher order schedulers can take advantage of
+    template < typename T >
+    void order_lrr( typename std::vector< T >& result_list,
+                    const typename std::vector< T >& input_list,
+                    const typename std::vector< T >::const_iterator& last_issued_from_input,
+                    unsigned num_warps_to_add );
+    
+    enum OrderingType 
+    {
+        // The item that issued last is prioritized first then the sorted result
+        // of the priority_function
+        ORDERING_GREEDY_THEN_PRIORITY_FUNC = 0,
+        // No greedy scheduling based on last to issue. Only the priority function determines
+        // priority
+        ORDERED_PRIORITY_FUNC_ONLY,
+        NUM_ORDERING,
+    };
+    template < typename U >
+    void order_by_priority( std::vector< U >& result_list,
+                            const typename std::vector< U >& input_list,
+                            const typename std::vector< U >::const_iterator& last_issued_from_input,
+                            unsigned num_warps_to_add,
+                            OrderingType age_ordering,
+                            bool (*priority_func)(U lhs, U rhs) );
+    static bool sort_warps_by_oldest_dynamic_id(shd_warp_t* lhs, shd_warp_t* rhs);
+
+    // Derived classes can override this function to populate
+    // m_supervised_warps with their scheduling policies
+    virtual void order_warps() = 0;
+
+protected:
+    virtual void do_on_warp_issued( unsigned warp_id,
+                                    unsigned num_issued,
+                                    const std::vector< shd_warp_t* >::const_iterator& prioritized_iter );
+    inline int get_sid() const;
 protected:
     shd_warp_t& warp(int i);
 
-    std::vector<int> supervised_warps;
-    int m_last_sup_id_issued;
+    // This is the prioritized warp list that is looped over each cycle to determine
+    // which warp gets to issue.
+    std::vector< shd_warp_t* > m_next_cycle_prioritized_warps;
+    // The m_supervised_warps list is all the warps this scheduler is supposed to
+    // arbitrate between.  This is useful in systems where there is more than
+    // one warp scheduler. In a single scheduler system, this is simply all
+    // the warps assigned to this core.
+    std::vector< shd_warp_t* > m_supervised_warps;
+    // This is the iterator pointer to the last supervised warp you issued
+    std::vector< shd_warp_t* >::const_iterator m_last_supervised_issued;
     shader_core_stats *m_stats;
     shader_core_ctx* m_shader;
     // these things should become accessors: but would need a bigger rearchitect of how shader_core_ctx interacts with its parts.
@@ -291,45 +365,89 @@ protected:
     register_set* m_sp_out;
     register_set* m_sfu_out;
     register_set* m_mem_out;
+
+    int m_id;
+};
+
+class lrr_scheduler : public scheduler_unit {
+public:
+	lrr_scheduler ( shader_core_stats* stats, shader_core_ctx* shader,
+                    Scoreboard* scoreboard, simt_stack** simt,
+                    std::vector<shd_warp_t>* warp,
+                    register_set* sp_out,
+                    register_set* sfu_out,
+                    register_set* mem_out,
+                    int id )
+	: scheduler_unit ( stats, shader, scoreboard, simt, warp, sp_out, sfu_out, mem_out, id ){}
+	virtual ~lrr_scheduler () {}
+	virtual void order_warps ();
+    virtual void done_adding_supervised_warps() {
+        m_last_supervised_issued = m_supervised_warps.end();
+    }
 };
 
-class LooseRoundRobbinScheduler : public scheduler_unit {
+class gto_scheduler : public scheduler_unit {
 public:
-	LooseRoundRobbinScheduler (shader_core_stats* stats, shader_core_ctx* shader,
-            Scoreboard* scoreboard, simt_stack** simt,
-            std::vector<shd_warp_t>* warp,
-            register_set* sp_out,
-            register_set* sfu_out,
-            register_set* mem_out)
-	: scheduler_unit (stats, shader, scoreboard, simt, warp, sp_out, sfu_out, mem_out){}
-	virtual ~LooseRoundRobbinScheduler () {}
-	virtual void cycle ();
+	gto_scheduler ( shader_core_stats* stats, shader_core_ctx* shader,
+                    Scoreboard* scoreboard, simt_stack** simt,
+                    std::vector<shd_warp_t>* warp,
+                    register_set* sp_out,
+                    register_set* sfu_out,
+                    register_set* mem_out,
+                    int id )
+	: scheduler_unit ( stats, shader, scoreboard, simt, warp, sp_out, sfu_out, mem_out, id ){}
+	virtual ~gto_scheduler () {}
+	virtual void order_warps ();
+    virtual void done_adding_supervised_warps() {
+        m_last_supervised_issued = m_supervised_warps.begin();
+    }
+
 };
 
 
-class TwoLevelScheduler : public scheduler_unit {
+class two_level_active_scheduler : public scheduler_unit {
 public:
-	TwoLevelScheduler (shader_core_stats* stats, shader_core_ctx* shader,
-            Scoreboard* scoreboard, simt_stack** simt,
-            std::vector<shd_warp_t>* warp,
-            register_set* sp_out,
-            register_set* sfu_out,
-            register_set* mem_out,
-            unsigned maw)
-	: scheduler_unit (stats, shader, scoreboard, simt, warp, sp_out, sfu_out, mem_out),
-	  activeWarps(),
-	  pendingWarps(){
-		maxActiveWarps = maw;
-	}
-	virtual ~TwoLevelScheduler () {}
-	virtual void cycle ();
-	virtual void add_supervised_warp_id(int i) {
-		pendingWarps.push_back(i);
+	two_level_active_scheduler ( shader_core_stats* stats, shader_core_ctx* shader,
+                          Scoreboard* scoreboard, simt_stack** simt,
+                          std::vector<shd_warp_t>* warp,
+                          register_set* sp_out,
+                          register_set* sfu_out,
+                          register_set* mem_out,
+                          int id,
+                          char* config_str )
+	: scheduler_unit ( stats, shader, scoreboard, simt, warp, sp_out, sfu_out, mem_out, id ),
+	  m_pending_warps()
+    {
+        int ret = sscanf( config_str,
+                          "two_level_active:%d:%d:%d",
+                          &m_max_active_warps,
+                          (int*)&m_inner_level_prioritization,
+                          (int*)&m_outer_level_prioritization );
+        assert( 3 == ret );
+    }
+	virtual ~two_level_active_scheduler () {}
+    virtual void order_warps();
+	void add_supervised_warp_id(int i) {
+        if ( m_next_cycle_prioritized_warps.size() < m_max_active_warps ) {
+            m_next_cycle_prioritized_warps.push_back( &warp(i) );
+        } else {
+		    m_pending_warps.push_back(&warp(i));
+        }
 	}
+    virtual void done_adding_supervised_warps() {
+        m_last_supervised_issued = m_supervised_warps.begin();
+    }
+
+protected:
+    virtual void do_on_warp_issued( unsigned warp_id,
+                                    unsigned num_issued,
+                                    const std::vector< shd_warp_t* >::const_iterator& prioritized_iter );
+
 private:
-	unsigned maxActiveWarps;
-	std::list<int> activeWarps;
-	std::list<int> pendingWarps;
+	std::deque< shd_warp_t* > m_pending_warps; 
+    scheduler_prioritization_type m_inner_level_prioritization;
+    scheduler_prioritization_type m_outer_level_prioritization;
+	unsigned m_max_active_warps;
 };
 
 
@@ -1158,6 +1276,7 @@ struct shader_core_config : public core_config
     //Shader core resources
     unsigned gpgpu_shader_registers;
     int gpgpu_warpdistro_shader;
+    int gpgpu_warp_issue_shader;
     unsigned gpgpu_num_reg_banks;
     bool gpgpu_reg_bank_use_warp_id;
     bool gpgpu_local_mem_map;
@@ -1324,7 +1443,7 @@ public:
         gpgpu_n_shmem_bank_access = (unsigned *)calloc(config->num_shader(), sizeof(unsigned));
 
         m_shader_dynamic_warp_issue_distro.resize( config->num_shader() );
-
+        m_shader_warp_slot_issue_distro.resize( config->num_shader() );
     }
 
     void new_grid()
@@ -1342,10 +1461,18 @@ public:
         return m_shader_dynamic_warp_issue_distro;
     }
 
+    const std::vector< std::vector<unsigned> >& get_warp_slot_issue() const
+    {
+        return m_shader_warp_slot_issue_distro;
+    }
+
 private:
     const shader_core_config *m_config;
     // Counts the instructions issued for each dynamic warp.
     std::vector< std::vector<unsigned> > m_shader_dynamic_warp_issue_distro;
+    std::vector<unsigned> m_last_shader_dynamic_warp_issue_distro;
+    std::vector< std::vector<unsigned> > m_shader_warp_slot_issue_distro;
+    std::vector<unsigned> m_last_shader_warp_slot_issue_distro;
 
     friend class power_stat_t;
     friend class shader_core_ctx;
@@ -1750,4 +1877,7 @@ private:
     simt_core_cluster *m_cluster;
 };
 
+
+inline int scheduler_unit::get_sid() const { return m_shader->get_sid(); }
+
 #endif /* SHADER_H */
diff --git a/src/gpgpu-sim/shader_trace.h b/src/gpgpu-sim/shader_trace.h
new file mode 100644
index 0000000..8a433cb
--- /dev/null
+++ b/src/gpgpu-sim/shader_trace.h
@@ -0,0 +1,74 @@
+// Copyright (c) 2009-2011, Tor M. Aamodt, Tim Rogers
+// George L. Yuan, Andrew Turner, Inderpreet Singh 
+// The University of British Columbia
+// All rights reserved.
+//
+// Redistribution and use in source and binary forms, with or without
+// modification, are permitted provided that the following conditions are met:
+//
+// Redistributions of source code must retain the above copyright notice, this
+// list of conditions and the following disclaimer.
+// Redistributions in binary form must reproduce the above copyright notice, this
+// list of conditions and the following disclaimer in the documentation and/or
+// other materials provided with the distribution.
+// Neither the name of The University of British Columbia nor the names of its
+// contributors may be used to endorse or promote products derived from this
+// software without specific prior written permission.
+//
+// THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND
+// ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
+// WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
+// DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE
+// FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+// DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
+// SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
+// CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
+// OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+// OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+
+#ifndef __SHADER_TRACE_H__
+#define __SHADER_TRACE_H__
+
+#include "../trace.h"
+
+
+#if TRACING_ON
+
+#define SHADER_PRINT_STR SIM_PRINT_STR "Core %d - "
+#define SCHED_PRINT_STR SHADER_PRINT_STR "Scheduler %d - "
+#define SHADER_DTRACE(x)  (DTRACE(x) && Trace::sampling_core == get_sid())
+
+// Intended to be called from inside components of a shader core.
+// Depends on a get_sid() function
+#define SHADER_DPRINTF(x, ...) do {\
+    if (SHADER_DTRACE(x)) {\
+        printf( SHADER_PRINT_STR,\
+                gpu_sim_cycle + gpu_tot_sim_cycle,\
+                Trace::trace_streams_str[Trace::x],\
+                get_sid() );\
+        printf(__VA_ARGS__);\
+    }\
+} while (0)
+
+// Intended to be called from inside a scheduler_unit.
+// Depends on a m_id member
+#define SCHED_DPRINTF(...) do {\
+    if (SHADER_DTRACE(WARP_SCHEDULER)) {\
+        printf( SCHED_PRINT_STR,\
+                gpu_sim_cycle + gpu_tot_sim_cycle,\
+                Trace::trace_streams_str[Trace::WARP_SCHEDULER],\
+                get_sid(),\
+                m_id );\
+        printf(__VA_ARGS__);\
+    }\
+} while (0)
+
+#else
+
+#define SHADER_DTRACE(x)  (false)
+#define SHADER_DPRINTF(x, ...) do {} while (0)
+#define SCHED_DPRINTF(x, ...) do {} while (0)
+
+#endif
+
+#endif
diff --git a/src/trace.cc b/src/trace.cc
new file mode 100644
index 0000000..05b9d6b
--- /dev/null
+++ b/src/trace.cc
@@ -0,0 +1,55 @@
+// Copyright (c) 2009-2013, Tor M. Aamodt, Timothy Rogers,
+// The University of British Columbia
+// All rights reserved.
+//
+// Redistribution and use in source and binary forms, with or without
+// modification, are permitted provided that the following conditions are met:
+//
+// Redistributions of source code must retain the above copyright notice, this
+// list of conditions and the following disclaimer.
+// Redistributions in binary form must reproduce the above copyright notice, this
+// list of conditions and the following disclaimer in the documentation and/or
+// other materials provided with the distribution.
+// Neither the name of The University of British Columbia nor the names of its
+// contributors may be used to endorse or promote products derived from this
+// software without specific prior written permission.
+//
+// THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND
+// ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
+// WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
+// DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE
+// FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+// DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
+// SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
+// CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
+// OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+// OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+
+#include "trace.h"
+#include "string.h"
+
+namespace Trace {
+
+
+#define TS_TUP_BEGIN(X) const char* trace_streams_str[] = {
+#define TS_TUP(X) #X
+#define TS_TUP_END(X) };
+#include "trace_streams.tup"
+#undef TS_TUP_BEGIN
+#undef TS_TUP
+#undef TS_TUP_END
+
+    bool enabled = false;
+    int sampling_core = 0;
+    bool trace_streams_enabled[NUM_TRACE_STREAMS] = {false};
+    const char* config_str;
+
+    void init()
+    {
+        for ( unsigned i = 0; i < NUM_TRACE_STREAMS; ++i ) {
+            if ( strstr( config_str, trace_streams_str[i] ) != NULL ) {
+                trace_streams_enabled[ i ] = true;
+            }
+        }
+    }
+} 
diff --git a/src/trace.h b/src/trace.h
new file mode 100644
index 0000000..c7b389e
--- /dev/null
+++ b/src/trace.h
@@ -0,0 +1,79 @@
+// Copyright (c) 2009-2013, Tor M. Aamodt, Timothy Rogers,
+// The University of British Columbia
+// All rights reserved.
+//
+// Redistribution and use in source and binary forms, with or without
+// modification, are permitted provided that the following conditions are met:
+//
+// Redistributions of source code must retain the above copyright notice, this
+// list of conditions and the following disclaimer.
+// Redistributions in binary form must reproduce the above copyright notice, this
+// list of conditions and the following disclaimer in the documentation and/or
+// other materials provided with the distribution.
+// Neither the name of The University of British Columbia nor the names of its
+// contributors may be used to endorse or promote products derived from this
+// software without specific prior written permission.
+//
+// THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND
+// ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
+// WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
+// DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE
+// FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+// DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
+// SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
+// CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
+// OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+// OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+
+// This file is inspired by the trace system in gem5.
+// This is a highly simplified version adpated for gpgpusim
+
+#ifndef __TRACE_H__
+#define __TRACE_H__
+
+extern unsigned long long  gpu_sim_cycle;
+extern unsigned long long  gpu_tot_sim_cycle;
+
+namespace Trace {
+
+#define TS_TUP_BEGIN(X) enum X {
+#define TS_TUP(X) X
+#define TS_TUP_END(X) };
+#include "trace_streams.tup"
+#undef TS_TUP_BEGIN
+#undef TS_TUP
+#undef TS_TUP_END
+
+    extern bool enabled;
+    extern int sampling_core;
+    extern const char* trace_streams_str[];
+    extern bool trace_streams_enabled[NUM_TRACE_STREAMS];
+    extern const char* config_str;
+
+    void init();
+
+} // namespace Trace
+
+
+#if TRACING_ON
+
+#define SIM_PRINT_STR "GPGPU-Sim Cycle %llu: %s - "
+#define DTRACE(x) ((Trace::trace_streams_enabled[Trace::x]) && Trace::enabled)
+#define DPRINTF(x, ...) do {\
+    if (DTRACE(x)) {\
+        printf( SIM_PRINT_STR,\
+                gpu_sim_cycle + gpu_tot_sim_cycle,\
+                Trace::trace_streams_str[Trace::x] );\
+        printf(__VA_ARGS__);\
+    }\
+} while (0)
+
+
+#else 
+
+#define DTRACE(x) (false)
+#define DPRINTF(x, ...) do {} while (0)
+
+#endif  
+
+#endif 
diff --git a/src/trace_streams.tup b/src/trace_streams.tup
new file mode 100644
index 0000000..fa6fd1e
--- /dev/null
+++ b/src/trace_streams.tup
@@ -0,0 +1,32 @@
+// Copyright (c) 2009 by Tor M. Aamodt, Tim Rogers and 
+// The University of British Columbia
+// All rights reserved.
+//
+// Redistribution and use in source and binary forms, with or without
+// modification, are permitted provided that the following conditions are met:
+//
+// Redistributions of source code must retain the above copyright notice, this
+// list of conditions and the following disclaimer.
+// Redistributions in binary form must reproduce the above copyright notice, this
+// list of conditions and the following disclaimer in the documentation and/or
+// other materials provided with the distribution.
+// Neither the name of The University of British Columbia nor the names of its
+// contributors may be used to endorse or promote products derived from this
+// software without specific prior written permission.
+//
+// THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND
+// ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
+// WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
+// DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE
+// FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+// DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
+// SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
+// CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
+// OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+// OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+
+TS_TUP_BEGIN( trace_streams_type )
+    TS_TUP( WARP_SCHEDULER ),
+    TS_TUP( SCOREBOARD ),
+    TS_TUP( NUM_TRACE_STREAMS )
+TS_TUP_END( trace_streams_type )
-- 
cgit v1.3