Tricky one, thinking out loud here:
Use an AreaOnAreaOverlayer, this will cut up areas where they overlap, calculate the area of each resulting polygon, then Dissolve them back together based on a "base" id. Have the Dissolver create a list and sort that on the area attribute.
Further to @redgeographics answer, after the AreaOnAreaOverlayer you need to explode the list to have one piece per overlap, otherwise the Dissolver will only have the overlap in one of the polygons, and the other(s) will have a gap.
For what it's worth, when I did this I found it more efficient to drop the geometry after calculating the area, and merge the list back on to the original features with a FeatureMerger, then to dissolve the pieces back together, though I did this back in 2014, and there have been improvements to the dissolver in the meantime.
Don't think you need to dissolve.
AreaOnAreaOverlayer.
1
Calculate area, sort by ID and Area attribute.
Sample 1st of each ID using a sampler grouped by ID.
merge the original objects by ID from the result of the sampler.
2.
After area calculation, aggregate by ID and candidate ID, sum the area . then again sort, now by ID and summed area.
Sample 1st ID and summed area.
Remerge original by ID.
No list building required either me thinks...