The Arcobjects one is the full Esri reader/writer with full geometry support, though to use it you'll also need Esri installed and a license available. The API option allows you to work with the GDB format with no Esri technology installed, BUT, it doesn't include full GDB feature support.

"FILEGDB" writer is a File Geodatabase (Esri API). GEODATABASE_FILE is a File Geodatabase (Arc Objects). Esri API is a 64 bit technology and runs in 64 bit FME, while Arc Objects only runs in 32 bit FME as it is an older technology. Esri API will skip writing "bad values" and Arc Objects will truncate them.

For more information check out these knowledgebase articles:

Comparison of FME readers and writers for SDE and Geodatabase: https://knowledge.safe.com/content/kbentry/19440/c...

Notes on FME and Esri Versions and Compatibility: https://knowledge.safe.com/articles/1517/notes-on-...