This is a case where dealing with PointClouds rather than rasters is much more efficient.
Add a counter to get an unique id on your buildings if they don't already have them. Set a NoDataValue on the Raster if it doesn't already have one.
Use a GeometryExtractor on the building to store the geometry as an attribute.
Clip the raster by buildings, with merge attributes set to yes.
Send the Clipped rasters to a PointCloudCombiner, grouping by the UniqueID
Use the PointCloudStatisticsCalculator to get the Mean of the z component.
Use a GeometryReplacer to restore the original building geometry.
You should now have your buildings with an addition attribute called z.mean.
It works correctly, thank you !
I wasn't on the right track.