You should probably be applying the border-radius on each individual button instead of on the whole btn-group.
<div class="btn-group" role="group" aria-label="...">
<button type="button" class="btn btn-default btn-rounded">Left</button>
<button type="button" class="btn btn-default btn-rounded">Middle</button>
<button type="button" class="btn btn-default btn-rounded">Right</button>
</div>
...and then the CSS class
.btn-rounded {
border-radius: 25px;
}