Can I use FME desktop with any third party server, lets say we are creating own instead of FME server will it work. If yes what modification or do safe provide the API.

You could, technically, do that. FME Desktop will run on a number of different OS'es. But depending on which part of the FME Server functionality to want to recreate it'll be a very tough job, especially when it comes to on-demand stuff.

In addition to that, you'll have to take a close look at the SLA as not all of the functionality of FME Server is allowed on the FME Desktop license.

So practically the answer would be "no, this is not feasible". If you have specific requirements do reach out to either Safe or your local Safe Partner, maybe something can be arranged.