As the documentation says:

The Douglas-Peucker algorithm will remove vertices which cause a deviation of less than the Generalization Tolerance, but the location of remaining vertices are not altered. Thus, this algorithm is good at reducing the number of points in a line, it is not very good at preserving the shape or the spatial relationship of the line relative to other entities. When used on polygons, the start point is never removed.

(bold emphasis is mine).

Since the algorithm begins processing at the starting vertex, then the second vertex, and so forth, I’m not at all surprised that changing the order of the vertices gives a different result.

​@s.jager by using a tolerance of zero, the shape is not altered and only the collinear vertices should disappear. The collinearity condition depends only on the previous and next vertex, so I am surprised that the order of the vertices affects the result. In other words, if given three vertices (P1, P2, P3) P2 is collinear, then so should be the case if the vertices with (P3, P2, P1).

For only three vertices in a line yes, you’re right, but most polygons have far more vertices….

If you think something is off with the software, the best thing to do is probably submit a ticket at Safe Software.

https://support.safe.com/hc/en-us/p/support

​@s.jager in the example above, the polygon has 484 vertices, but as far as I know, the result of Douglas' algorithm should be independent of the geometry orientation. I will follow ​@nielsgerrits’s suggestion and submit a ticket.

I suspect the issue with the “Vertices order sensitive Generalizer” is related to how FME optimizes accuracy.  At a very basic level, you can calculate the vertical distance of a point from a line using the attribute manager in FME.  If you do this with your data without doing anything to improve the computational accuracy than you will get 4 verticies that are colinear regardless of direction. 

If you then make a lazy attempt at improving the computational accuracy by subtracting the minimum x and y value from every coordinate than you will find that no points are colinear.  Hence, you can speculate that FME could potentialy use a forward looking algorthim that considers the next so many values to improve computational accuracy and this could be in an interative or recursive approach and then you will get the observed directional dependence.  

The attached FME workspace calculates the perpendicular distance of a point from a line for each of your verticies, and also shows the difference if you improve the computational accurace by subctracting the minimum x and y.