Mark,

- When you change a thread's subject, have the system send out an email
to the group saying 'Hey, this is the new subject, start replying to
this email instead'

Ick. And it likely wouldn't work - members are apt to read messages in order and may have already replied to the thread before seeing this message.

- We just disable the subject line check for threads that have had their
subject's changed. Everything will work fine except if someone decides
they want to start a new thread by replying to an existing message and
changing the subject.

That's probably a reasonable compromise. Maybe the disable could be for a limited duration (a few days) - replies later than that may as well start a new thread.

- Somehow we keep track of any subject changes and use those when
figuring out if a subject line has changed. This is really complicated.

Maybe only the most recent prior subject line, rather than the full history of changes.

Either way, this points back to a need for a mechanism to explicitly break and connect threads. Er, topics. Provides a means to repair any unfortunate consequences of whatever mechanism is chosen for handling Subject edits.
https://groups.io/org/groupsio/beta/message/502

-- Shal