Of course we don't recommend doing large version jumps from 1.5.x (especially if not the last 1.5.x) to 1.6.8 for a reason. You jumped over three years of updates on one go.
There have been lots of changes and one includes a database encoding change to support utf8mb4 encoding (the actual real utf8 MySQL didn't formerly). That may affect here or some wrong settings. Or your server does not support utf8 or doesn't have the multibyte or iconv extension. Setup will tell.
Found the bug, the email headers are not properly separated in the send_mail member function of class-utf8, email headers requires "\r\n" not just "\n":
lib/class-utf8.php
class-utf8:end_mail
line 713:
if ($additional_headers != '')
$additional_headers .= "\r\n";
$additional_headers .=
"Mime-Version: 1.0\r\n" .
"Content-Type: text/plain; charset=UTF-8\r\n" .
"Content-Transfer-Encoding: base64";