Saludos podrias crear de la siguiente manera:
$(document).ready(function() {
$('.modal').each(function() {
$this = $(this);
//al abrir play video
$this.on('shown.bs.modal', function() {
toggleVideo('playVideo', $(this));
});
//al cerrar modal pause video
$this.on('hidden.bs.modal', function(){
toggleVideo('pauseVideo', $(this));
})
});
function toggleVideo(state, div) {
var iframe = div.find("iframe")[0].contentWindow;
iframe.postMessage('{"event":"command","func":"' + state + '","args":""}', '*');
}
});
<!doctype html>
<html lang="en">
<head>
<!-- Required meta tags -->
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no">
<!-- Bootstrap CSS -->
<link rel="stylesheet" href="https://stackpath.bootstrapcdn.com/bootstrap/4.3.1/css/bootstrap.min.css" integrity="sha384-ggOyR0iXCbMQv3Xipma34MD+dH/1fQ784/j6cY/iJTQUOhcWr7x9JvoRxT2MZw1T" crossorigin="anonymous">
<title>Hello, world!</title>
</head>
<body>
<!-- Button trigger modal -->
<button type="button" class="btn btn-primary" data-toggle="modal" data-target="#exampleModal">
Abrir Video
</button>
<!-- Modal -->
<div class="modal fade" id="exampleModal" tabindex="-1" role="dialog" aria-labelledby="exampleModalLabel" aria-hidden="true">
<div class="modal-dialog" role="document">
<div class="modal-content">
<div class="modal-header">
<h5 class="modal-title" id="exampleModalLabel">Video Youtube</h5>
<button type="button" class="close" data-dismiss="modal" aria-label="Close">
<span aria-hidden="true">×</span>
</button>
</div>
<div class="modal-body">
<iframe width="200" height="150" class="youtubeplayer" id="youtubeplayer" src="https://www.youtube.com/embed/34Na4j8AVgA?list=PLDcnymzs18LWrKzHmzrGH1JzLBqrHi3xQ&enablejsapi=1" frameborder="0" allowfullscreen></iframe>
</div>
<div class="modal-footer">
<button type="button" class="btn btn-danger" data-dismiss="modal">Salir</button>
</div>
</div>
</div>
</div>
<!-- Optional JavaScript -->
<!-- jQuery first, then Popper.js, then Bootstrap JS -->
<script src="https://code.jquery.com/jquery-3.3.1.slim.min.js" integrity="sha384-q8i/X+965DzO0rT7abK41JStQIAqVgRVzpbzo5smXKp4YfRvH+8abtTE1Pi6jizo" crossorigin="anonymous"></script>
<script src="https://cdnjs.cloudflare.com/ajax/libs/popper.js/1.14.7/umd/popper.min.js" integrity="sha384-UO2eT0CpHqdSJQ6hJty5KVphtPhzWj9WO1clHTMGa3JDZwrnQq4sF86dIHNDz0W1" crossorigin="anonymous"></script>
<script src="https://stackpath.bootstrapcdn.com/bootstrap/4.3.1/js/bootstrap.min.js" integrity="sha384-JjSmVgyd0p3pXB1rRibZUAYoIIy6OrQ6VrjIEaFf/nJGzIxFDsf4x0xIM+B07jRM" crossorigin="anonymous"></script>
</body>
</html>
Lo hace basicamente el Script es al momento de abrir el modal reproduce el video con:
toggleVideo('playVideo', $(this));
y al cerrar el modal lo pone en pausa:
toggleVideo('pauseVideo', $(this));
la funcion toggleVideo recibe dos parametros el evento deseado(playVideo, pauseVideo, stopVideo) y el elemento donde se encuentra el video.
tambien puedes probarlo desde aca: Jsfiddle dado que stack overflow bloquea el contenido de video de Youtube.
Espero te sirva y suerte..!!