Question

FME Server performance reporting

N
Badge +4

Hi,

I'm looking at ways of assessing capacity of our fme server instance(s), and as well as regular things like cpu/ram capacity on the server I'm interested in the application specific metrics.

In particular does anyone have a workbench/tool that will query an FME server 2015 repository and report the max number of engines running in a defined time period, lets say hourly. (I'm interested in the concurrency of engines, so I can reduce the chance of queuing, particuarly when I have active time dependant jobs.)

If any one has any other stat gathering workbenches, I'd be interested in them too. I have some that query the repository and work out some stats per workbench, avg/max run times, success/fail counts that sort of stuff.

Thanks in advance,

Nick

10 replies

david_r
Rank
Userlevel 4

FME Server 2016.1 and on has a new Dashboard functionality that you can activate (it is disabled by default), it will give you some out-of-the-box statistics such as average run times, etc. You can find some info about it here.

If you have an older version of FME Server or you need something a bit more specific, your best bet is to query the PostgreSQL repository database, where you will find all the details about all the jobs (queued, running and finished/failed). Using those tables it is relatively easy to make very targeted reports and statistics, such as the examples you mention.

Connection details for the PostgreSQL instance can be found here: https://knowledge.safe.com/questions/21761/connecting-to-fme-server-postgresql-database-repos.html

N
Badge +4

Thanks for the quick reply david_r,

I'm looking forward to seeing the new version, however it'll be awhile (years) before I'm able to finance the upgrade of our current version of FME Server.

Yes, I'm querying the repository (oracle in our case) for some of the workbench stats I refer to, which works well.

Getting the max concurrent engine count per hour is trickier than I imagined though, so was wondering whether anyone had already done this?

(the real bad news is that I think I DID work this out! I remember scratching my head about it a lot, but I can't find my work! so looking for a shortcut really.)

david_r
Rank
Userlevel 4

Thanks for the quick reply david_r,

I'm looking forward to seeing the new version, however it'll be awhile (years) before I'm able to finance the upgrade of our current version of FME Server.

Yes, I'm querying the repository (oracle in our case) for some of the workbench stats I refer to, which works well.

Getting the max concurrent engine count per hour is trickier than I imagined though, so was wondering whether anyone had already done this?

(the real bad news is that I think I DID work this out! I remember scratching my head about it a lot, but I can't find my work! so looking for a shortcut really.)

Maybe look inside the table job_history and analyze the fields date_started and date_finished.

The maximum number of overlapping jobs at any given time will be the max concurrent engine count.

todd_davis
Rank
Userlevel 1
Badge +12

Here is an example of some of the things we are doing (this is a couple of years old now), if anything is on interest.

https://www.webmaps.co.nz/fmeserver/Reporting/20160719.html

Workbench is pretty ugly (and deprecated google charts required ugly workarounds), but you can have the workbench.

N
Badge +4

Here is an example of some of the things we are doing (this is a couple of years old now), if anything is on interest.

https://www.webmaps.co.nz/fmeserver/Reporting/20160719.html

Workbench is pretty ugly (and deprecated google charts required ugly workarounds), but you can have the workbench.

 

@todd_davis thanks for this. I have something very similar but running internally. You're missing the question i'm asking specifically though, but luckily i have now found the workbench i started this awhile ago so will give it a spring clean and perhaps post it if anyone is interested.

 

 

Is there a place to share workbenches on this site btw? I know you can share transformers through the store.
N
Badge +4

I managed to track down the work I did before on this and got the answers I needed.

The SQL to do this is actually quite tricky and complex (or at least mine was), so I came up with plan b, which fits very nicely in my spatially orientated head so I thought I'd share the methodology as the workbench is then easy to knock up.

1) get all the records from the fme_job_hostory table.

2) for each record convert (or copy and convert) the start and end time to epoch seconds (%s in dateformatter)

3) convert each feature into a line using the start time and end time epoch seconds for x setting y to 0

4) generate a baseline by using the summarystatistics to get the min start time and max end time, then generating a line. (you need this to get the times when sever isn't doing anything)

5) use the lineonlineoverlayer to intersect all the lines - the 'overlaps' attribute now contains the number of engines running at any time, and you can use the lengthcalculator to return the number of seconds this concurrency lasted for. (remember to subtract the baseline from the overlaps count)

6) I then output this to excel, using a 'state_changed' date/time which I got by using the coordinate extractor to get the first point of each intersected line, then converting the epoch seconds to a normal date/time using python (why won't the dateformatter do this btw?)

7) a bit of pivot table magic in excel then got me some nice graphs showing peak concurrency by hour, peak concurrency by day etc

itay
Rank
Badge +16

Very nice! and similar to what is now done in de FME Server dashboards workspaces that are installed with FME server 2016.1

N
Badge +4

if only I didn't cost me £100K and 18 months worth of effort to get my IT supplier to update our software. (it costs £30K and 6 months to update the desktop install!)

Looking forward to experminting with FME cloud

itay
Rank
Badge +16

if only I didn't cost me £100K and 18 months worth of effort to get my IT supplier to update our software. (it costs £30K and 6 months to update the desktop install!)

Looking forward to experminting with FME cloud

well why dont you? doesnt cost you a dime to try and you can play around for awhile before the spending money you are given is finished....

 

 

todd_davis
Rank
Userlevel 1
Badge +12

I managed to track down the work I did before on this and got the answers I needed.

The SQL to do this is actually quite tricky and complex (or at least mine was), so I came up with plan b, which fits very nicely in my spatially orientated head so I thought I'd share the methodology as the workbench is then easy to knock up.

1) get all the records from the fme_job_hostory table.

2) for each record convert (or copy and convert) the start and end time to epoch seconds (%s in dateformatter)

3) convert each feature into a line using the start time and end time epoch seconds for x setting y to 0

4) generate a baseline by using the summarystatistics to get the min start time and max end time, then generating a line. (you need this to get the times when sever isn't doing anything)

5) use the lineonlineoverlayer to intersect all the lines - the 'overlaps' attribute now contains the number of engines running at any time, and you can use the lengthcalculator to return the number of seconds this concurrency lasted for. (remember to subtract the baseline from the overlaps count)

6) I then output this to excel, using a 'state_changed' date/time which I got by using the coordinate extractor to get the first point of each intersected line, then converting the epoch seconds to a normal date/time using python (why won't the dateformatter do this btw?)

7) a bit of pivot table magic in excel then got me some nice graphs showing peak concurrency by hour, peak concurrency by day etc

Sounds like what I do i as well....turning time into spatial data tpo find overlaps etcs

 

 

Reply


Community Stats

28,727
Posts
109,521
Replies
36,566
Members
Terms & ConditionsCookie settings
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.