Merging

//depot/gpgpu_sim_research/fermi_tim/...

to //depot/gpgpu_sim_research/fermi/...

Integrating CLs up to 15295. Descriptions of these CL's are included.

***
A couple changes to aeriel-vision for warp issue plot support
***
More arielvision changes to support the variable-entry length stacked bar chart
***
Properly printing the right resolution of dynamic warp ids
***.
Generalized the scheduler code and added detailed statistics for which warps issue each cycle.
   Verified the execution of the LRR scheduler - still have to get the two level scheduler to work.
***
Implementing the 2lvl scehduler has it has been originally coded.
   LRR on both the inner and outer levels
***
Adding in a debug tracing system to GPGPU-Sim.
   I am sick of writing debug code - then having to comment out, ifdef out or delete it to checkin.
   This also allows for print streams so the user can decided which traces they would like to see.
   Every print in GPGPU-Sim should go through this system - then it will be really easy to only get the information you want and more importantly people will (a) write and (b) checkin code that actually profiles what they are building.
   Reading tracefiles is superiour in many ways to single stepping since you can print the world and just vet the logfile for what you need.
   This also fascilitates advice from the Debugging Rules! book which states that you should never throw away a debugging tool.  Having debug prints that don't get thrown away is big.
***
   Allowing the trace to be specified in the Make.
   Run Make TRACE=0 to compile the code without any traces
***
   Allowing prints from the performance sim to get the actual ptx instruction text
***
Getting the two level scheduler to actaully work...
   What is released in fermi does not work at all - it effectively performs "static warp limit" from my CCWS paper.
   Warps are never demoted from the active list since the functionality checking to see if they are waiting on a longop is completly broken.
   Maybe if the original author had access to the tracing functions this would not have happened.
   The islongop test was completely broken.  It did not mark the register as used, it marked the register number in the instruction as used.
   For example if this instruction was creating a long op:
   ld r6 [r1]
   It would mark register 0 as waiting for a long op (since it is register 0 of the two registers in this instruction), not register 6.
   Additionally, whenever ANY instruction from a warp releases registers, ALL the longops being tracked for this warp get cleared....
   The only way anyone ever thought this worked is if they did not test it....
***
Reworking the warp schedulers to share common code.
   Making the GTX480 use gto by default. I am not sure wht they really use, but it really can't be LRR.
   Also adding in a new file for custom shared trace defines. These are useful when you want a print that has some
   additional criteria or information printed.
   Verified that the schedulers all work to a first order based on traces.
***
Making it so you can run the stats collection scripts from any directory.
   Also allow the caller to specify a stats file instead of just assume its always the same one

[git-p4: depot-paths = "//depot/gpgpu_sim_research/fermi/distribution/": change = 15296]

1 files changed, 517 insertions, 289 deletions

diff --git a/src/gpgpu-sim/shader.cc b/src/gpgpu-sim/shader.cc
index 7a822b2..9d974db 100644
--- a/src/gpgpu-sim/shader.cc
+++ b/src/gpgpu-sim/shader.cc
@@ -45,9 +45,11 @@
 #include "icnt_wrapper.h"
 #include <string.h>
 #include <limits.h>
+#include "shader_trace.h"
 
 #define PRIORITIZE_MSHR_OVER_WB 1
 #define MAX(a,b) (((a)>(b))?(a):(b))
+    
 
 /////////////////////////////////////////////////////////////////////////////
 
@@ -72,170 +74,213 @@ shader_core_ctx::shader_core_ctx( class gpgpu_sim *gpu,
    : m_barriers( config->max_warps_per_shader, config->max_cta_per_core ),
      m_dynamic_warp_id(0)
 {
-   m_kernel = NULL;
-   m_gpu = gpu;
-   m_cluster = cluster;
-   m_config = config;
-   m_memory_config = mem_config;
-   m_stats = stats;
-   unsigned warp_size=config->warp_size;
-
-   m_sid = shader_id;
-   m_tpc = tpc_id;
-
-   m_pipeline_reg.reserve(N_PIPELINE_STAGES);
-   for (int j = 0; j<N_PIPELINE_STAGES; j++) {
-      m_pipeline_reg.push_back(register_set(m_config->pipe_widths[j],pipeline_stage_name_decode[j]));
-   }
-
-   m_threadState = (thread_ctx_t*) calloc(sizeof(thread_ctx_t), config->n_thread_per_shader);
-   m_thread = (ptx_thread_info**) calloc(sizeof(ptx_thread_info*), config->n_thread_per_shader);
-
-   m_not_completed = 0;
-   m_active_threads.reset();
-   m_n_active_cta = 0;
-   for (unsigned i = 0; i<MAX_CTA_PER_SHADER; i++  ) 
-      m_cta_status[i]=0;
-   for (unsigned i = 0; i<config->n_thread_per_shader; i++) {
-      m_thread[i]= NULL;
-      m_threadState[i].m_cta_id = -1;
-      m_threadState[i].m_active = false;
-   }
-   
-   // m_icnt = new shader_memory_interface(this,cluster);
+    m_kernel = NULL;
+    m_gpu = gpu;
+    m_cluster = cluster;
+    m_config = config;
+    m_memory_config = mem_config;
+    m_stats = stats;
+    unsigned warp_size=config->warp_size;
+    
+    m_sid = shader_id;
+    m_tpc = tpc_id;
+    
+    m_pipeline_reg.reserve(N_PIPELINE_STAGES);
+    for (int j = 0; j<N_PIPELINE_STAGES; j++) {
+        m_pipeline_reg.push_back(register_set(m_config->pipe_widths[j],pipeline_stage_name_decode[j]));
+    }
+    
+    m_threadState = (thread_ctx_t*) calloc(sizeof(thread_ctx_t), config->n_thread_per_shader);
+    m_thread = (ptx_thread_info**) calloc(sizeof(ptx_thread_info*), config->n_thread_per_shader);
+    
+    m_not_completed = 0;
+    m_active_threads.reset();
+    m_n_active_cta = 0;
+    for ( unsigned i = 0; i<MAX_CTA_PER_SHADER; i++ ) 
+        m_cta_status[i]=0;
+    for (unsigned i = 0; i<config->n_thread_per_shader; i++) {
+        m_thread[i]= NULL;
+        m_threadState[i].m_cta_id = -1;
+        m_threadState[i].m_active = false;
+    }
+    
+    // m_icnt = new shader_memory_interface(this,cluster);
     if ( m_config->gpgpu_perfect_mem ) {
         m_icnt = new perfect_memory_interface(this,cluster);
     } else {
         m_icnt = new shader_memory_interface(this,cluster);
     }
-   m_mem_fetch_allocator = new shader_core_mem_fetch_allocator(shader_id,tpc_id,mem_config);
-
-   // fetch
-   m_last_warp_fetched = 0;
-
-   #define STRSIZE 1024
-   char name[STRSIZE];
-   snprintf(name, STRSIZE, "L1I_%03d", m_sid);
-   m_L1I = new read_only_cache( name,m_config->m_L1I_config,m_sid,get_shader_instruction_cache_id(),m_icnt,IN_L1I_MISS_QUEUE);
-
-   m_warp.resize(m_config->max_warps_per_shader, shd_warp_t(this, warp_size));
-   initilizeSIMTStack(config->max_warps_per_shader,this->get_config()->warp_size);
-   m_scoreboard = new Scoreboard(m_sid, m_config->max_warps_per_shader);
-
-   //scedulers
-   //must currently occur after all inputs have been initialized.
-   std::string sched_config = m_config->gpgpu_scheduler_string;
-   bool tlsched = sched_config.find("tl") != std::string::npos;
-   int tlmaw;
-   if (tlsched){
-	   sscanf(m_config->gpgpu_scheduler_string, "tl:%d", &tlmaw);
-   }
-   for (int i = 0; i < m_config->gpgpu_num_sched_per_core; i++) {
-	   //###CONFIGURATION
-	   if (tlsched){
-		   schedulers.push_back(new TwoLevelScheduler(m_stats,this,m_scoreboard,m_simt_stack,&m_warp,
-				   &m_pipeline_reg[ID_OC_SP],
-				   &m_pipeline_reg[ID_OC_SFU],
-				   &m_pipeline_reg[ID_OC_MEM],
-				   tlmaw));
-	   } else {
-		   schedulers.push_back(new LooseRoundRobbinScheduler(m_stats,this,m_scoreboard,m_simt_stack,&m_warp,
-				   &m_pipeline_reg[ID_OC_SP],
-				   &m_pipeline_reg[ID_OC_SFU],
-				   &m_pipeline_reg[ID_OC_MEM]));
-	   }
-   }
-   for (unsigned i = 0; i < m_warp.size(); i++) {
-       //distribute i's evenly though schedulers;
-       schedulers[i%m_config->gpgpu_num_sched_per_core]->add_supervised_warp_id(i);
-   }
-
-   //op collector configuration
-   enum { SP_CUS, SFU_CUS, MEM_CUS, GEN_CUS };
-   m_operand_collector.add_cu_set(SP_CUS, m_config->gpgpu_operand_collector_num_units_sp, m_config->gpgpu_operand_collector_num_out_ports_sp);
-   m_operand_collector.add_cu_set(SFU_CUS, m_config->gpgpu_operand_collector_num_units_sfu, m_config->gpgpu_operand_collector_num_out_ports_sfu);
-   m_operand_collector.add_cu_set(MEM_CUS, m_config->gpgpu_operand_collector_num_units_mem, m_config->gpgpu_operand_collector_num_out_ports_mem);
-   m_operand_collector.add_cu_set(GEN_CUS, m_config->gpgpu_operand_collector_num_units_gen, m_config->gpgpu_operand_collector_num_out_ports_gen);
-
-   opndcoll_rfu_t::port_vector_t in_ports;
-   opndcoll_rfu_t::port_vector_t out_ports;
-   opndcoll_rfu_t::uint_vector_t cu_sets;
-   for (unsigned i = 0; i < m_config->gpgpu_operand_collector_num_in_ports_sp; i++) {
-       in_ports.push_back(&m_pipeline_reg[ID_OC_SP]);
-       out_ports.push_back(&m_pipeline_reg[OC_EX_SP]);
-       cu_sets.push_back((unsigned)SP_CUS);
-       cu_sets.push_back((unsigned)GEN_CUS);
-       m_operand_collector.add_port(in_ports,out_ports,cu_sets);
-       in_ports.clear(),out_ports.clear(),cu_sets.clear();
-   }
-
-   for (unsigned i = 0; i < m_config->gpgpu_operand_collector_num_in_ports_sfu; i++) {
-       in_ports.push_back(&m_pipeline_reg[ID_OC_SFU]);
-       out_ports.push_back(&m_pipeline_reg[OC_EX_SFU]);
-       cu_sets.push_back((unsigned)SFU_CUS);
-       cu_sets.push_back((unsigned)GEN_CUS);
-       m_operand_collector.add_port(in_ports,out_ports,cu_sets);
-       in_ports.clear(),out_ports.clear(),cu_sets.clear();
-   }
-
-   for (unsigned i = 0; i < m_config->gpgpu_operand_collector_num_in_ports_mem; i++) {
-       in_ports.push_back(&m_pipeline_reg[ID_OC_MEM]);
-       out_ports.push_back(&m_pipeline_reg[OC_EX_MEM]);
-       cu_sets.push_back((unsigned)MEM_CUS);
-       cu_sets.push_back((unsigned)GEN_CUS);                       
-       m_operand_collector.add_port(in_ports,out_ports,cu_sets);
-       in_ports.clear(),out_ports.clear(),cu_sets.clear();
-   }   
-
-
-   for (unsigned i = 0; i < m_config->gpgpu_operand_collector_num_in_ports_gen; i++) {
-       in_ports.push_back(&m_pipeline_reg[ID_OC_SP]);
-       in_ports.push_back(&m_pipeline_reg[ID_OC_SFU]);
-       in_ports.push_back(&m_pipeline_reg[ID_OC_MEM]);
-       out_ports.push_back(&m_pipeline_reg[OC_EX_SP]);
-       out_ports.push_back(&m_pipeline_reg[OC_EX_SFU]);
-       out_ports.push_back(&m_pipeline_reg[OC_EX_MEM]);
-       cu_sets.push_back((unsigned)GEN_CUS);   
-       m_operand_collector.add_port(in_ports,out_ports,cu_sets);
-       in_ports.clear(),out_ports.clear(),cu_sets.clear();
-   }
-
-   m_operand_collector.init( m_config->gpgpu_num_reg_banks, this );
-
-   // execute
-   m_num_function_units = m_config->gpgpu_num_sp_units + m_config->gpgpu_num_sfu_units + 1; // sp_unit, sfu, ldst_unit
-   //m_dispatch_port = new enum pipeline_stage_name_t[ m_num_function_units ];
-   //m_issue_port = new enum pipeline_stage_name_t[ m_num_function_units ];
-
-   //m_fu = new simd_function_unit*[m_num_function_units];
-
-   for (int k = 0; k < m_config->gpgpu_num_sp_units; k++) {
-       m_fu.push_back(new sp_unit( &m_pipeline_reg[EX_WB], m_config, this ));
-       m_dispatch_port.push_back(ID_OC_SP);
-       m_issue_port.push_back(OC_EX_SP);
-   }
-
-   for (int k = 0; k < m_config->gpgpu_num_sfu_units; k++) {
-       m_fu.push_back(new sfu( &m_pipeline_reg[EX_WB], m_config, this ));
-       m_dispatch_port.push_back(ID_OC_SFU);
-       m_issue_port.push_back(OC_EX_SFU);
-   }
-
-   m_ldst_unit = new ldst_unit( m_icnt, m_mem_fetch_allocator, this, &m_operand_collector, m_scoreboard, config, mem_config, stats, shader_id, tpc_id );
-   m_fu.push_back(m_ldst_unit);
-   m_dispatch_port.push_back(ID_OC_MEM);
-   m_issue_port.push_back(OC_EX_MEM);
-
-   assert(m_num_function_units == m_fu.size() and m_fu.size() == m_dispatch_port.size() and m_fu.size() == m_issue_port.size());
-
-   //there are as many result buses as the width of the EX_WB stage
-   num_result_bus = config->pipe_widths[EX_WB];
-   for(unsigned i=0; i<num_result_bus; i++){
-	   this->m_result_bus.push_back(new std::bitset<MAX_ALU_LATENCY>());
-   }
-
-   m_last_inst_gpu_sim_cycle = 0;
-   m_last_inst_gpu_tot_sim_cycle = 0;
+    m_mem_fetch_allocator = new shader_core_mem_fetch_allocator(shader_id,tpc_id,mem_config);
+    
+    // fetch
+    m_last_warp_fetched = 0;
+    
+    #define STRSIZE 1024
+    char name[STRSIZE];
+    snprintf(name, STRSIZE, "L1I_%03d", m_sid);
+    m_L1I = new read_only_cache( name,m_config->m_L1I_config,m_sid,get_shader_instruction_cache_id(),m_icnt,IN_L1I_MISS_QUEUE);
+    
+    m_warp.resize(m_config->max_warps_per_shader, shd_warp_t(this, warp_size));
+    initilizeSIMTStack(config->max_warps_per_shader,this->get_config()->warp_size);
+    m_scoreboard = new Scoreboard(m_sid, m_config->max_warps_per_shader);
+    
+    //scedulers
+    //must currently occur after all inputs have been initialized.
+    std::string sched_config = m_config->gpgpu_scheduler_string;
+    const concrete_scheduler scheduler = sched_config.find("lrr") != std::string::npos ?
+                                         CONCRETE_SCHEDULER_LRR :
+                                         sched_config.find("two_level_active") != std::string::npos ?
+                                         CONCRETE_SCHEDULER_TWO_LEVEL_ACTIVE :
+                                         sched_config.find("gto") != std::string::npos ?
+                                         CONCRETE_SCHEDULER_GTO :
+                                         NUM_CONCRETE_SCHEDULERS;
+    assert ( scheduler != NUM_CONCRETE_SCHEDULERS );
+    
+    for (int i = 0; i < m_config->gpgpu_num_sched_per_core; i++) {
+        switch( scheduler )
+        {
+            case CONCRETE_SCHEDULER_LRR:
+                schedulers.push_back(
+                    new lrr_scheduler( m_stats,
+                                       this,
+                                       m_scoreboard,
+                                       m_simt_stack,
+                                       &m_warp,
+                                       &m_pipeline_reg[ID_OC_SP],
+                                       &m_pipeline_reg[ID_OC_SFU],
+                                       &m_pipeline_reg[ID_OC_MEM],
+                                       i
+                                     )
+                );
+                break;
+            case CONCRETE_SCHEDULER_TWO_LEVEL_ACTIVE:
+                schedulers.push_back(
+                    new two_level_active_scheduler( m_stats,
+                                                    this,
+                                                    m_scoreboard,
+                                                    m_simt_stack,
+                                                    &m_warp,
+                                                    &m_pipeline_reg[ID_OC_SP],
+                                                    &m_pipeline_reg[ID_OC_SFU],
+                                                    &m_pipeline_reg[ID_OC_MEM],
+                                                    i,
+                                                    config->gpgpu_scheduler_string
+                                                  )
+                );
+                break;
+            case CONCRETE_SCHEDULER_GTO:
+                schedulers.push_back(
+                    new gto_scheduler( m_stats,
+                                       this,
+                                       m_scoreboard,
+                                       m_simt_stack,
+                                       &m_warp,
+                                       &m_pipeline_reg[ID_OC_SP],
+                                       &m_pipeline_reg[ID_OC_SFU],
+                                       &m_pipeline_reg[ID_OC_MEM],
+                                       i
+                                     )
+                );
+                break;
+            default:
+                abort();
+        };
+    }
+    
+    for (unsigned i = 0; i < m_warp.size(); i++) {
+        //distribute i's evenly though schedulers;
+        schedulers[i%m_config->gpgpu_num_sched_per_core]->add_supervised_warp_id(i);
+    }
+    for ( int i = 0; i < m_config->gpgpu_num_sched_per_core; ++i ) {
+        schedulers[i]->done_adding_supervised_warps();
+    }
+    
+    //op collector configuration
+    enum { SP_CUS, SFU_CUS, MEM_CUS, GEN_CUS };
+    m_operand_collector.add_cu_set(SP_CUS, m_config->gpgpu_operand_collector_num_units_sp, m_config->gpgpu_operand_collector_num_out_ports_sp);
+    m_operand_collector.add_cu_set(SFU_CUS, m_config->gpgpu_operand_collector_num_units_sfu, m_config->gpgpu_operand_collector_num_out_ports_sfu);
+    m_operand_collector.add_cu_set(MEM_CUS, m_config->gpgpu_operand_collector_num_units_mem, m_config->gpgpu_operand_collector_num_out_ports_mem);
+    m_operand_collector.add_cu_set(GEN_CUS, m_config->gpgpu_operand_collector_num_units_gen, m_config->gpgpu_operand_collector_num_out_ports_gen);
+    
+    opndcoll_rfu_t::port_vector_t in_ports;
+    opndcoll_rfu_t::port_vector_t out_ports;
+    opndcoll_rfu_t::uint_vector_t cu_sets;
+    for (unsigned i = 0; i < m_config->gpgpu_operand_collector_num_in_ports_sp; i++) {
+        in_ports.push_back(&m_pipeline_reg[ID_OC_SP]);
+        out_ports.push_back(&m_pipeline_reg[OC_EX_SP]);
+        cu_sets.push_back((unsigned)SP_CUS);
+        cu_sets.push_back((unsigned)GEN_CUS);
+        m_operand_collector.add_port(in_ports,out_ports,cu_sets);
+        in_ports.clear(),out_ports.clear(),cu_sets.clear();
+    }
+    
+    for (unsigned i = 0; i < m_config->gpgpu_operand_collector_num_in_ports_sfu; i++) {
+        in_ports.push_back(&m_pipeline_reg[ID_OC_SFU]);
+        out_ports.push_back(&m_pipeline_reg[OC_EX_SFU]);
+        cu_sets.push_back((unsigned)SFU_CUS);
+        cu_sets.push_back((unsigned)GEN_CUS);
+        m_operand_collector.add_port(in_ports,out_ports,cu_sets);
+        in_ports.clear(),out_ports.clear(),cu_sets.clear();
+    }
+    
+    for (unsigned i = 0; i < m_config->gpgpu_operand_collector_num_in_ports_mem; i++) {
+        in_ports.push_back(&m_pipeline_reg[ID_OC_MEM]);
+        out_ports.push_back(&m_pipeline_reg[OC_EX_MEM]);
+        cu_sets.push_back((unsigned)MEM_CUS);
+        cu_sets.push_back((unsigned)GEN_CUS);                       
+        m_operand_collector.add_port(in_ports,out_ports,cu_sets);
+        in_ports.clear(),out_ports.clear(),cu_sets.clear();
+    }   
+    
+    
+    for (unsigned i = 0; i < m_config->gpgpu_operand_collector_num_in_ports_gen; i++) {
+        in_ports.push_back(&m_pipeline_reg[ID_OC_SP]);
+        in_ports.push_back(&m_pipeline_reg[ID_OC_SFU]);
+        in_ports.push_back(&m_pipeline_reg[ID_OC_MEM]);
+        out_ports.push_back(&m_pipeline_reg[OC_EX_SP]);
+        out_ports.push_back(&m_pipeline_reg[OC_EX_SFU]);
+        out_ports.push_back(&m_pipeline_reg[OC_EX_MEM]);
+        cu_sets.push_back((unsigned)GEN_CUS);   
+        m_operand_collector.add_port(in_ports,out_ports,cu_sets);
+        in_ports.clear(),out_ports.clear(),cu_sets.clear();
+    }
+    
+    m_operand_collector.init( m_config->gpgpu_num_reg_banks, this );
+    
+    // execute
+    m_num_function_units = m_config->gpgpu_num_sp_units + m_config->gpgpu_num_sfu_units + 1; // sp_unit, sfu, ldst_unit
+    //m_dispatch_port = new enum pipeline_stage_name_t[ m_num_function_units ];
+    //m_issue_port = new enum pipeline_stage_name_t[ m_num_function_units ];
+    
+    //m_fu = new simd_function_unit*[m_num_function_units];
+    
+    for (int k = 0; k < m_config->gpgpu_num_sp_units; k++) {
+        m_fu.push_back(new sp_unit( &m_pipeline_reg[EX_WB], m_config, this ));
+        m_dispatch_port.push_back(ID_OC_SP);
+        m_issue_port.push_back(OC_EX_SP);
+    }
+    
+    for (int k = 0; k < m_config->gpgpu_num_sfu_units; k++) {
+        m_fu.push_back(new sfu( &m_pipeline_reg[EX_WB], m_config, this ));
+        m_dispatch_port.push_back(ID_OC_SFU);
+        m_issue_port.push_back(OC_EX_SFU);
+    }
+    
+    m_ldst_unit = new ldst_unit( m_icnt, m_mem_fetch_allocator, this, &m_operand_collector, m_scoreboard, config, mem_config, stats, shader_id, tpc_id );
+    m_fu.push_back(m_ldst_unit);
+    m_dispatch_port.push_back(ID_OC_MEM);
+    m_issue_port.push_back(OC_EX_MEM);
+    
+    assert(m_num_function_units == m_fu.size() and m_fu.size() == m_dispatch_port.size() and m_fu.size() == m_issue_port.size());
+    
+    //there are as many result buses as the width of the EX_WB stage
+    num_result_bus = config->pipe_widths[EX_WB];
+    for(unsigned i=0; i<num_result_bus; i++){
+        this->m_result_bus.push_back(new std::bitset<MAX_ALU_LATENCY>());
+    }
+    
+    m_last_inst_gpu_sim_cycle = 0;
+    m_last_inst_gpu_tot_sim_cycle = 0;
 }
 
 void shader_core_ctx::reinit(unsigned start_thread, unsigned end_thread, bool reset_not_completed ) 
@@ -433,6 +478,10 @@ void shader_core_stats::event_warp_issued( unsigned s_id, unsigned warp_id, unsi
             m_shader_dynamic_warp_issue_distro[ s_id ].resize(dynamic_warp_id + 1);
         }
         ++m_shader_dynamic_warp_issue_distro[ s_id ][ dynamic_warp_id ];
+        if ( m_shader_warp_slot_issue_distro[ s_id ].size() <= warp_id ) {
+            m_shader_warp_slot_issue_distro[ s_id ].resize(warp_id + 1);
+        }
+        ++m_shader_warp_slot_issue_distro[ s_id ][ warp_id ];
     }
 }
 
@@ -457,6 +506,47 @@ void shader_core_stats::visualizer_print( gzFile visualizer_file )
     }
     gzprintf(visualizer_file,"\n");
 
+    // warp issue breakdown
+    unsigned sid = m_config->gpgpu_warp_issue_shader;
+    gzprintf(visualizer_file, "WarpIssueSlotBreakdown:");
+    unsigned count = 0;
+    unsigned warp_id_issued_sum = 0;
+    for ( std::vector<unsigned>::const_iterator iter = m_shader_warp_slot_issue_distro[ sid ].begin();
+          iter != m_shader_warp_slot_issue_distro[ sid ].end(); iter++, count++ ) {
+        unsigned diff = count < m_last_shader_warp_slot_issue_distro.size() ?
+                        *iter - m_last_shader_warp_slot_issue_distro[ count ] :
+                        *iter;
+        gzprintf( visualizer_file, " %d", diff );
+        warp_id_issued_sum += diff;
+    }
+    m_last_shader_warp_slot_issue_distro = m_shader_warp_slot_issue_distro[ sid ];
+    gzprintf(visualizer_file,"\n");
+
+    #define DYNAMIC_WARP_PRINT_RESOLUTION 32
+    unsigned total_issued_this_resolution = 0;
+    unsigned dynamic_id_issued_sum = 0;
+    count = 0;
+    gzprintf(visualizer_file, "WarpIssueDynamicIdBreakdown:");
+    for ( std::vector<unsigned>::const_iterator iter = m_shader_dynamic_warp_issue_distro[ sid ].begin();
+          iter != m_shader_dynamic_warp_issue_distro[ sid ].end(); iter++, count++ ) {
+        unsigned diff = count < m_last_shader_dynamic_warp_issue_distro.size() ?
+                        *iter - m_last_shader_dynamic_warp_issue_distro[ count ] :
+                        *iter;
+        total_issued_this_resolution += diff;
+        if ( ( count + 1 ) % DYNAMIC_WARP_PRINT_RESOLUTION == 0 ) {
+            gzprintf( visualizer_file, " %d", total_issued_this_resolution );
+            dynamic_id_issued_sum += total_issued_this_resolution;
+            total_issued_this_resolution = 0;
+        }
+    }
+    if ( count % DYNAMIC_WARP_PRINT_RESOLUTION != 0 ) {
+        gzprintf( visualizer_file, " %d", total_issued_this_resolution );
+        dynamic_id_issued_sum += total_issued_this_resolution;
+    }
+    m_last_shader_dynamic_warp_issue_distro = m_shader_dynamic_warp_issue_distro[ sid ];
+    gzprintf(visualizer_file,"\n");
+    assert( warp_id_issued_sum == dynamic_id_issued_sum );
+
     // overall cache miss rates
     gzprintf(visualizer_file, "gpgpu_n_cache_bkconflict: %d\n", gpgpu_n_cache_bkconflict);
     gzprintf(visualizer_file, "gpgpu_n_shmem_bkconflict: %d\n", gpgpu_n_shmem_bkconflict);     
@@ -631,15 +721,113 @@ shd_warp_t& scheduler_unit::warp(int i){
     return (*m_warp)[i];
 }
 
-void LooseRoundRobbinScheduler::cycle()
+
+/**
+ * A general function to order things in a Loose Round Robin way. The simplist use of this
+ * function would be to implement a loose RR scheduler between all the warps assigned to this core.
+ * A more sophisticated usage would be to order a set of "fetch groups" in a RR fashion.
+ * In the first case, the templated class variable would be a simple unsigned int representing the
+ * warp_id.  In the 2lvl case, T could be a struct or a list representing a set of warp_ids.
+ * @param result_list: The resultant list the caller wants returned.  This list is cleared and then populated
+ *                     in a loose round robin way
+ * @param input_list: The list of things that should be put into the result_list. For a simple scheduler
+ *                    this can simply be the m_supervised_warps list.
+ * @param last_issued_from_input:  An iterator pointing the last member in the input_list that issued.
+ *                                 Since this function orders in a RR fashion, the object pointed
+ *                                 to by this iterator will be last in the prioritization list
+ * @param num_warps_to_add: The number of warps you want the scheudler to pick between this cycle.
+ *                          Normally, this will be all the warps availible on the core, i.e.
+ *                          m_supervised_warps.size(). However, a more sophisticated scheduler may wish to
+ *                          limit this number. If the number if < m_supervised_warps.size(), then only
+ *                          the warps with highest RR priority will be placed in the result_list.
+ */
+template < class T >
+void scheduler_unit::order_lrr( std::vector< T >& result_list,
+                                const typename std::vector< T >& input_list,
+                                const typename std::vector< T >::const_iterator& last_issued_from_input,
+                                unsigned num_warps_to_add )
+{
+    assert( num_warps_to_add <= input_list.size() );
+    result_list.clear();
+    typename std::vector< T >::const_iterator iter
+        = ( last_issued_from_input ==  input_list.end() ) ? input_list.begin()
+                                                          : last_issued_from_input + 1;
+
+    for ( unsigned count = 0;
+          count < num_warps_to_add;
+          ++iter, ++count) {
+        if ( iter ==  input_list.end() ) {
+            iter = input_list.begin();
+        }
+        result_list.push_back( *iter );
+    }
+}
+
+/**
+ * A general function to order things in an priority-based way.
+ * The core usage of the function is similar to order_lrr.
+ * The explanation of the additional parameters (beyond order_lrr) explains the further extensions.
+ * @param ordering: An enum that determines how the age function will be treated in prioritization
+ *                  see the definition of OrderingType.
+ * @param priority_function: This function is used to sort the input_list.  It is passed to stl::sort as
+ *                           the sorting fucntion. So, if you wanted to sort a list of integer warp_ids
+ *                           with the oldest warps having the most priority, then the priority_function
+ *                           would compare the age of the two warps.
+ */
+template < class T >
+void scheduler_unit::order_by_priority( std::vector< T >& result_list,
+                                        const typename std::vector< T >& input_list,
+                                        const typename std::vector< T >::const_iterator& last_issued_from_input,
+                                        unsigned num_warps_to_add,
+                                        OrderingType ordering,
+                                        bool (*priority_func)(T lhs, T rhs) )
+{
+    assert( num_warps_to_add <= input_list.size() );
+    result_list.clear();
+    typename std::vector< T > temp = input_list;
+
+
+    if ( ORDERING_GREEDY_THEN_PRIORITY_FUNC == ordering ) {
+        T greedy_value = *last_issued_from_input;
+        result_list.push_back( greedy_value );
+
+        std::sort( temp.begin(), temp.end(), priority_func );
+        typename std::vector< T >::iterator iter = temp.begin();
+        for ( unsigned count = 0; count < num_warps_to_add; ++count, ++iter ) {
+            if ( *iter != greedy_value ) {
+                result_list.push_back( *iter );
+            }
+        }
+    } else if ( ORDERED_PRIORITY_FUNC_ONLY == ordering ) {
+        std::sort( temp.begin(), temp.end(), priority_func );
+        typename std::vector< T >::iterator iter = temp.begin();
+        for ( unsigned count = 0; count < num_warps_to_add; ++count, ++iter ) {
+            result_list.push_back( *iter );
+        }
+    } else {
+        fprintf( stderr, "Unknown ordering - %d\n", ordering );
+        abort();
+    }
+}
+
+void scheduler_unit::cycle()
 {
+    SCHED_DPRINTF( "scheduler_unit::cycle()\n" );
     bool valid_inst = false;  // there was one warp with a valid instruction to issue (didn't require flush due to control hazard)
     bool ready_inst = false;  // of the valid instructions, there was one not waiting for pending register writes
     bool issued_inst = false; // of these we issued one
 
-    for ( unsigned i=0; i < supervised_warps.size(); i++ ) {
-        unsigned supervised_id = (m_last_sup_id_issued+1+i) % supervised_warps.size();
-        unsigned warp_id = supervised_warps[supervised_id];
+    order_warps();
+    for ( std::vector< shd_warp_t* >::const_iterator iter = m_next_cycle_prioritized_warps.begin();
+          iter != m_next_cycle_prioritized_warps.end();
+          iter++ ) {
+        // Don't consider warps that are not yet valid
+        if ( (*iter) == NULL || (*iter)->done_exit() ) {
+            continue;
+        }
+        SCHED_DPRINTF( "Testing (warp_id %u, dynamic_warp_id %u)\n",
+                       (*iter)->get_warp_id(), (*iter)->get_dynamic_warp_id() );
+        unsigned warp_id = (*iter)->get_warp_id();
         unsigned checked=0;
         unsigned issued=0;
         unsigned max_issue = m_shader->m_config->gpgpu_max_insn_issue_per_warp;
@@ -649,15 +837,22 @@ void LooseRoundRobbinScheduler::cycle()
             bool warp_inst_issued = false;
             unsigned pc,rpc;
             m_simt_stack[warp_id]->get_pdom_stack_top_info(&pc,&rpc);
+            SCHED_DPRINTF( "Warp (warp_id %u, dynamic_warp_id %u) has valid instruction (%s)\n",
+                           (*iter)->get_warp_id(), (*iter)->get_dynamic_warp_id(),
+                           ptx_get_insn_str( pc).c_str() );
             if( pI ) {
                 assert(valid);
                 if( pc != pI->pc ) {
+                    SCHED_DPRINTF( "Warp (warp_id %u, dynamic_warp_id %u) control hazard instruction flush\n",
+                                   (*iter)->get_warp_id(), (*iter)->get_dynamic_warp_id() );
                     // control hazard
                     warp(warp_id).set_next_pc(pc);
                     warp(warp_id).ibuffer_flush();
                 } else {
                     valid_inst = true;
                     if ( !m_scoreboard->checkCollision(warp_id, pI) ) {
+                        SCHED_DPRINTF( "Warp (warp_id %u, dynamic_warp_id %u) passes scoreboard\n",
+                                       (*iter)->get_warp_id(), (*iter)->get_dynamic_warp_id() );
                         ready_inst = true;
                         const active_mask_t &active_mask = m_simt_stack[warp_id]->get_active_mask();
                         assert( warp(warp_id).inst_in_pipeline() );
@@ -686,19 +881,40 @@ void LooseRoundRobbinScheduler::cycle()
                                 }
                             } 
                         }
+                    } else {
+                        SCHED_DPRINTF( "Warp (warp_id %u, dynamic_warp_id %u) fails scoreboard\n",
+                                       (*iter)->get_warp_id(), (*iter)->get_dynamic_warp_id() );
                     }
                 }
             } else if( valid ) {
                // this case can happen after a return instruction in diverged warp
+               SCHED_DPRINTF( "Warp (warp_id %u, dynamic_warp_id %u) return from diverged warp flush\n",
+                              (*iter)->get_warp_id(), (*iter)->get_dynamic_warp_id() );
                warp(warp_id).set_next_pc(pc);
                warp(warp_id).ibuffer_flush();
             }
-            if(warp_inst_issued)
-               warp(warp_id).ibuffer_step();
+            if(warp_inst_issued) {
+                SCHED_DPRINTF( "Warp (warp_id %u, dynamic_warp_id %u) issued %u instructions\n",
+                               (*iter)->get_warp_id(),
+                               (*iter)->get_dynamic_warp_id(),
+                               issued );
+                do_on_warp_issued( warp_id, issued, iter );
+            }
             checked++;
         }
         if ( issued ) {
-            m_last_sup_id_issued=supervised_id;
+            // This might be a bit inefficient, but we need to maintain
+            // two ordered list for proper scheduler execution.
+            // We could remove the need for this loop by associating a
+            // supervised_is index with each entry in the m_next_cycle_prioritized_warps
+            // vector. For now, just run through until you find the right warp_id
+            for ( std::vector< shd_warp_t* >::const_iterator supervised_iter = m_supervised_warps.begin();
+                  supervised_iter != m_supervised_warps.end();
+                  ++supervised_iter ) {
+                if ( *iter == *supervised_iter ) {
+                    m_last_supervised_issued = supervised_iter;
+                }
+            }
             break;
         } 
     }
@@ -712,138 +928,110 @@ void LooseRoundRobbinScheduler::cycle()
         m_stats->shader_cycle_distro[2]++; // pipeline stalled
 }
 
-void TwoLevelScheduler::cycle() {
-	//Move waiting warps to pendingWarps
-	for (	std::list<int>::iterator iter = activeWarps.begin();
-			iter != activeWarps.end();
-			iter ++) {
-		bool waiting = warp(*iter).waiting();
-		for (int i=0; i<4; i++){
-			const warp_inst_t* inst = warp(*iter).ibuffer_next_inst();
-			//Is the instruction waiting on a long operation?
-			if ( inst && inst->in[i] > 0 && this->m_scoreboard->islongop(*iter, inst->in[i])){
-				waiting = true;
-			}
-		}
-
-		if(waiting){
-			pendingWarps.push_back(*iter);
-			activeWarps.erase(iter);
-			break;
-		}
-	}
+void scheduler_unit::do_on_warp_issued( unsigned warp_id,
+                                        unsigned num_issued,
+                                        const std::vector< shd_warp_t* >::const_iterator& prioritized_iter )
+{
+    m_stats->event_warp_issued( m_shader->get_sid(),
+                                warp_id,
+                                num_issued,
+                                warp(warp_id).get_dynamic_warp_id() );
+    warp(warp_id).ibuffer_step();
+}
 
-	//If there is space in activeWarps, try to find ready warps in pendingWarps
-	if (this->activeWarps.size() < maxActiveWarps){
-		for (	std::list<int>::iterator iter = pendingWarps.begin();
-				iter != pendingWarps.end();
-				iter++){
-			if(!warp(*iter).waiting()){
-				activeWarps.push_back(*iter);
-				pendingWarps.erase(iter);
-				break;
-			}
-		}
-	}
+bool scheduler_unit::sort_warps_by_oldest_dynamic_id(shd_warp_t* lhs, shd_warp_t* rhs)
+{
+    if (rhs && lhs) {
+        return lhs->get_dynamic_warp_id() < rhs->get_dynamic_warp_id();
+    } else {
+        return lhs < rhs;
+    }
+}
 
-	//Do the scheduling only from activeWarps
-	//If you schedule an instruction, move it to the end of the list
+void lrr_scheduler::order_warps()
+{
+    order_lrr( m_next_cycle_prioritized_warps,
+               m_supervised_warps,
+               m_last_supervised_issued,
+               m_supervised_warps.size() );
+}
 
-	bool valid_inst = false;  // there was one warp with a valid instruction to issue (didn't require flush due to control hazard)
-	bool ready_inst = false;  // of the valid instructions, there was one not waiting for pending register writes
-	bool issued_inst = false; // of these we issued one
+void gto_scheduler::order_warps()
+{
+    order_by_priority( m_next_cycle_prioritized_warps,
+                       m_supervised_warps,
+                       m_last_supervised_issued,
+                       m_supervised_warps.size(),
+                       ORDERING_GREEDY_THEN_PRIORITY_FUNC,
+                       scheduler_unit::sort_warps_by_oldest_dynamic_id );
+}
 
-	for (	std::list<int>::iterator warp_id = activeWarps.begin();
-			warp_id != activeWarps.end();
-			warp_id++) {
-		unsigned checked=0;
-		unsigned issued=0;
-		unsigned max_issue = m_shader->m_config->gpgpu_max_insn_issue_per_warp;
-		while(!warp(*warp_id).waiting() && !warp(*warp_id).ibuffer_empty() && (checked < max_issue) && (checked <= issued) && (issued < max_issue) ) {
-			const warp_inst_t *pI = warp(*warp_id).ibuffer_next_inst();
-			bool valid = warp(*warp_id).ibuffer_next_valid();
-			bool warp_inst_issued = false;
-			unsigned pc,rpc;
-			m_simt_stack[*warp_id]->get_pdom_stack_top_info(&pc,&rpc);
-			if( pI ) {
-				assert(valid);
-				if( pc != pI->pc ) {
-					// control hazard
-					warp(*warp_id).set_next_pc(pc);
-					warp(*warp_id).ibuffer_flush();
-				} else {
-					valid_inst = true;
-					if ( !m_scoreboard->checkCollision(*warp_id, pI) ) {
-						ready_inst = true;
-						const active_mask_t &active_mask = m_simt_stack[*warp_id]->get_active_mask();
-						assert( warp(*warp_id).inst_in_pipeline() );
-						if ( (pI->op == LOAD_OP) || (pI->op == STORE_OP) || (pI->op == MEMORY_BARRIER_OP) ) {
-							if( m_mem_out->has_free() ) {
-								m_shader->issue_warp(*m_mem_out,pI,active_mask,*warp_id);
-								issued++;
-								issued_inst=true;
-								warp_inst_issued = true;
-								// Move it to pendingWarps
-								unsigned currwarp = *warp_id;
-								activeWarps.erase(warp_id);
-								activeWarps.push_back(currwarp);
-							}
-						} else {
-							bool sp_pipe_avail = m_sp_out->has_free();
-							bool sfu_pipe_avail = m_sfu_out->has_free();
-							if( sp_pipe_avail && (pI->op != SFU_OP) ) {
-								// always prefer SP pipe for operations that can use both SP and SFU pipelines
-								m_shader->issue_warp(*m_sp_out,pI,active_mask,*warp_id);
-								issued++;
-								issued_inst=true;
-								warp_inst_issued = true;
-								//Move it to end of the activeWarps
-								unsigned currwarp = *warp_id;
-								activeWarps.erase(warp_id);
-								activeWarps.push_back(currwarp);
-							} else if ( (pI->op == SFU_OP) || (pI->op == ALU_SFU_OP) ) {
-								if( sfu_pipe_avail ) {
-									m_shader->issue_warp(*m_sfu_out,pI,active_mask,*warp_id);
-									issued++;
-									issued_inst=true;
-									warp_inst_issued = true;
-									//Move it to end of the activeWarps
-									unsigned currwarp = *warp_id;
-									activeWarps.erase(warp_id);
-									activeWarps.push_back(currwarp);
+void
+two_level_active_scheduler::do_on_warp_issued( unsigned warp_id,
+                                               unsigned num_issued,
+                                               const std::vector< shd_warp_t* >::const_iterator& prioritized_iter )
+{
+    scheduler_unit::do_on_warp_issued( warp_id, num_issued, prioritized_iter );
+    if ( SCHEDULER_PRIORITIZATION_LRR == m_inner_level_prioritization ) {
+        std::vector< shd_warp_t* > new_active; 
+        order_lrr( new_active,
+                   m_next_cycle_prioritized_warps,
+                   prioritized_iter,
+                   m_next_cycle_prioritized_warps.size() );
+        m_next_cycle_prioritized_warps = new_active;
+    } else {
+        fprintf( stderr,
+                 "Unimplemented m_inner_level_prioritization: %d\n",
+                 m_inner_level_prioritization );
+        abort();
+    }
+}
 
-								}
-							}
-						}
-					}
-				}
-			} else if( valid ) {
-				// this case can happen after a return instruction in diverged warp
-				warp(*warp_id).set_next_pc(pc);
-				warp(*warp_id).ibuffer_flush();
+void two_level_active_scheduler::order_warps()
+{
+	//Move waiting warps to m_pending_warps
+    unsigned num_demoted = 0;
+	for (	std::vector< shd_warp_t* >::iterator iter = m_next_cycle_prioritized_warps.begin();
+			iter != m_next_cycle_prioritized_warps.end(); ) {
+		bool waiting = (*iter)->waiting();
+		for (int i=0; i<4; i++){
+			const warp_inst_t* inst = (*iter)->ibuffer_next_inst();
+			//Is the instruction waiting on a long operation?
+			if ( inst && inst->in[i] > 0 && this->m_scoreboard->islongop((*iter)->get_warp_id(), inst->in[i])){
+				waiting = true;
 			}
-			if(warp_inst_issued)
-				warp(*warp_id).ibuffer_step();
-			checked++;
-		}
-		if ( issued ) {
-			break;
 		}
+
+		if( waiting ) {
+			m_pending_warps.push_back(*iter);
+			iter = m_next_cycle_prioritized_warps.erase(iter);
+            SCHED_DPRINTF( "DEMOTED warp_id=%d, dynamic_warp_id=%d\n",
+                           (*iter)->get_warp_id(),
+                           (*iter)->get_dynamic_warp_id() );
+            ++num_demoted;
+		} else {
+            ++iter;
+        }
 	}
 
-    // tgrogers - fixing a warning about unused variables at the top of this fucntion...
-    // This scheduler has A LOT of copied code from the LRR scheduler.
-    // TODO - this thing really needs to be re-written in a modular way.
-    // For now, to get rid of the warnings I am including the stats epilogue
-    // from the original scheduler function that the author of this function forgot to copy/paste
-    //
-    // issue stall statistics:
-    if( !valid_inst ) 
-        m_stats->shader_cycle_distro[0]++; // idle or control hazard
-    else if( !ready_inst ) 
-        m_stats->shader_cycle_distro[1]++; // waiting for RAW hazards (possibly due to memory) 
-    else if( !issued_inst ) 
-        m_stats->shader_cycle_distro[2]++; // pipeline stalled
+	//If there is space in m_next_cycle_prioritized_warps, promote the next m_pending_warps
+    unsigned num_promoted = 0;
+    if ( SCHEDULER_PRIORITIZATION_SRR == m_outer_level_prioritization ) {
+        while ( m_next_cycle_prioritized_warps.size() < m_max_active_warps ) {
+            m_next_cycle_prioritized_warps.push_back(m_pending_warps.front());
+            m_pending_warps.pop_front();
+            SCHED_DPRINTF( "PROMOTED warp_id=%d, dynamic_warp_id=%d\n",
+                           (m_next_cycle_prioritized_warps.back())->get_warp_id(),
+                           (m_next_cycle_prioritized_warps.back())->get_dynamic_warp_id() );
+            ++num_promoted;
+    	}
+    } else {
+        fprintf( stderr,
+                 "Unimplemented m_outer_level_prioritization: %d\n",
+                 m_outer_level_prioritization );
+        abort();
+    }
+    assert( num_promoted == num_demoted );
 }
 
 void shader_core_ctx::read_operands()
@@ -1727,6 +1915,46 @@ void gpgpu_sim::shader_print_runtime_stat( FILE *fout )
 }
 
 
+void gpgpu_sim::shader_print_scheduler_stat( FILE* fout, bool print_dynamic_info ) const
+{
+    // Print out the stats from the sampling shader core
+    const unsigned scheduler_sampling_core = m_shader_config->gpgpu_warp_issue_shader;
+    #define STR_SIZE 55
+    char name_buff[ STR_SIZE ];
+    name_buff[ STR_SIZE - 1 ] = '\0';
+    const std::vector< unsigned >& distro
+        = print_dynamic_info ?
+          m_shader_stats->get_dynamic_warp_issue()[ scheduler_sampling_core ] :
+          m_shader_stats->get_warp_slot_issue()[ scheduler_sampling_core ];
+    if ( print_dynamic_info ) {
+        snprintf( name_buff, STR_SIZE - 1, "dynamic_warp_id" );
+    } else {
+        snprintf( name_buff, STR_SIZE - 1, "warp_id" );
+    }
+    fprintf( fout,
+             "Shader %d %s issue ditsribution:\n",
+             scheduler_sampling_core,
+             name_buff );
+    const unsigned num_warp_ids = distro.size();
+    // First print out the warp ids
+    fprintf( fout, "%s:\n", name_buff );
+    for ( unsigned warp_id = 0;
+          warp_id < num_warp_ids;
+          ++warp_id  ) {
+        fprintf( fout, "%d, ", warp_id );
+    }
+
+    fprintf( fout, "\ndistro:\n" );
+    // Then print out the distribution of instuctions issued
+    for ( std::vector< unsigned >::const_iterator iter = distro.begin();
+          iter != distro.end();
+          iter++ ) {
+        fprintf( fout, "%d, ", *iter );
+    }
+    fprintf( fout, "\n" );
+}
+
+
 void gpgpu_sim::shader_print_l1_miss_stat( FILE *fout ) const
 {
    unsigned total_d1_misses = 0, total_d1_accesses = 0;