I don't know of any automated way to do unit testing in FME.
For integration tests of an FME workspace in a system/application, parameterization is key, and concatenated private parameters are your friend.
I like to isolate the different sections in embedded custom transformers. That way it is easy to identify and test a specific function, and it makes you more conscious of implicit assumptions about attributes. ie the attributes are not visible unless explicitly exposed.
When developing, recorder/players can be quite helpful to test specific sections.
I would recommend a test/production environment (but I suppose you have something like that in place already) where you do all the workspace development and testing seperate from the production data. Only when a workspace works perfectly in test does it get promoted to production.
Another thing you can do is use WorkspaceRunners to run individual workspaces with tests. If a child workspace fails you'll get a feature through the WorkspaceRunner's failed port which allows for easy logging. That would be more useful for a situation where you have a number of business-critical workspaces and want to run a new FME build through all of those tests before rolling it out.
Be very vigilant about retaining older versions of the workspaces (and, for that matter, back up installers of old FME versions too). Source control can help with this although I don't use that myself.