Here's the layout I want:

!Layout

I'm having some trouble converting it into code.

I've been thinking about bootstrap but I have elements that go across columns and rows and i'm just not sure how to code it and keep everything in the 6x4 grid I have created for the layout.

Another problem in this regard is the responsiveness. The boxes in the grid are supposed to stack when the viewport is made shorter. however this also creates problems when elements use a different number of columns.

Any ideas on how I should approach this?

I suggest looking reading up on how the bootstrap grid works and how you can nest rows within rows.

Here's a rough mockup of how you could acheive your design with bootstrap

<!-- begin snippet: js hide: false --> <!-- language: lang-css -->
.mygrid {
  text-align: center;
}
.mygrid div {
  border: 1px solid grey;
}
<!-- language: lang-html -->
<link href="//cdnjs.cloudflare.com/ajax/libs/twitter-bootstrap/3.3.2/css/bootstrap.css" rel="stylesheet"/>

<div class="container">
  <div class="mygrid">

    <!-- Top Half -->
    <div class="row">
      <div class="col-sm-2">
        <div class="row">1</div>
        <div class="row">2</div>
      </div>  
      <div class="col-sm-4">
        3<br/>
        3.5
      </div> 
      <div class="col-sm-2">
        <div class="row">4</div>
        <div class="row">5</div>
      </div>  
      <div class="col-sm-2">
        <div class="row">6</div>
        <div class="row">7</div>
      </div>  
      <div class="col-sm-2">
        <div class="row">8</div>
        <div class="row">9</div>
      </div>  
    </div>
    
    <!-- Bottom Half -->
    <div class="row">
      <div class="col-sm-6">
        <div class="row">
          <div class="col-sm-8">10</div>
          <div class="col-sm-4">11</div>
        </div>
        <div class="row">
          <div class="col-sm-4">12</div>
          <div class="col-sm-8">13</div>            
        </div>
      </div>  
      <div class="col-sm-4">
        14<br/>
        14.5
      </div> 
      <div class="col-sm-2">
        <div class="row">15</div>
        <div class="row">16</div>
      </div> 

    </div>

  </div>
</div>
<!-- end snippet -->

I added the .mygrid class just to help set some styles on the grid, but it's not necessary.