Finally found a set of LTO options that work for the vast majority of Gentoo packages, unlike my previous configuration where I had well over 300 exceptions and counting.
This is with GCC 6 as well, which hasn't even hit unstable yet!
I do need to submit a patch for sysklogd though
By "vast majority", I mean I currently have only two packages that have LTO disabled so far out of ~250 packages that I've rebuilt. So maybe 20 at most by the time I've gotten through all 1250 installed packages?