Plan 9 troff might work! It works with utf8 out of the box[0], and while I haven't used it for complex math typesetting, there is a command (eqn [1]) that was developed for it. I'd recommend Ali Rudi's port (neatroff [2][3]) for a minimalist implementation. There's also Heirloom Documentation Tools [4] which is an implementation of *roff-and-friends that uses Knuth's paragraph-at-once algorithm (instead of the original line-wise one) for typesetting, plus some other interesting features.
The authors of eqn wrote a paper about it: "Typesetting Mathematics" by Brian Kernighan and Lorinda Cherry. Kernighan also wrote two manuals (one in 1976 with a revision in 1992, and one in 2007 with updates for the Plan 9 version). [5].
The authors of eqn wrote a paper about it: "Typesetting Mathematics" by Brian Kernighan and Lorinda Cherry. Kernighan also wrote two manuals (one in 1976 with a revision in 1992, and one in 2007 with updates for the Plan 9 version). [5].
[0] utf8 was developed by Ken Thompson and Rob Pike during the creation of Plan9. The entire OS is compatible. Story here: http://doc.cat-v.org/bell_labs/utf-8_history
[1] http://man.cat-v.org/plan_9/1/eqn
[2] https://github.com/aligrudi/neatroff
[3] PDF manual for neatroff: http://litcave.rudi.ir/neatroff.pdf
[4] https://n-t-roff.github.io/heirloom/doctools.html
[5] These (and more) can be found here: http://www.kohala.com/start/troff/troff.html