From my experience, RasterStatsCalc is pretty quick. So therefore I'm imagining that the slowness is at the clipping?
I've recently been working on a piece of work involving 10000s of rasters that have required similar workflows to above. What I have found worked well in my case (lots of small rasters - if you're dealing with large rasters, may be worth tiling them into smaller areas) is building your clip areas, then use the FeatureReader to read in rasters that intersect your clip areas. Then perform the clip.
This means each clip is only dealing with an area of rasters that are in close proximity to it. Should be much faster