Safety is not the issue. The first, third, fifth, seventh, ninth, (and eleventh, for the 11-shot) chambers belong to one player, the second, fourth, sixth, eighth, and tenth chambers belong to the other.
I like Genmax's argument that the first player with the 11-shot has more chambers that have a chance of having a bullet than the second player, so there's a slightly better chance of winning by going first with 11-shot.