Solved

How to filter jobs by queue? How to sort the grid by 1 column or multiple columns?

N
Badge

I have set up job routing rules and I want to test that the routing rule is working fine. I have added CPU%, Memory Peak Usage, and Queue in Display Column on the grid, but there seems to be a lack of functionality on the grid or I am missing something basic.

 

Usecase: I want to see all the jobs which finish within 2 mins, use < 20% CPU, use <2GB memory. and want to make sure they all were assigned Micro queue.

 

-> I am trying to filter by queue but filtering by queue is not an option.

-> Grid is configurable to hide and show columns, but I can't sort by columns or add any kind of calculated field in the grid.

 

Is there any easy way to export this data to excel? I tried dashboards but they don't have the fields I am looking for, maybe if someone who understands the database schema of the FME server can guide us I can get help from our FME Author to adjust the workspace/dashboard to include these extra fields & filters to make administration little bit easy.

 

I am open to any other path or recommendations.

icon

Best answer by hollyatsafe 31 March 2022, 02:16

View original

12 replies

H
Badge +2

Hi @namanpatel​ ,

It is not currently possible to filter/sort the Jobs Completed page based on the job statistics fields. I think this would be a great enhancement and would recommend you post an idea.

As a workaround, I'd recommend you use the REST API GET /transformations/jobs/completed to fetch this data and then build a workspace to create the dashboard you desire. Here is a tutorial with some examples: https://community.safe.com/s/article/Monitoring-FME-Server-Job-Activity-using-the-REST-API

 

N
Badge

Hi @namanpatel​ ,

It is not currently possible to filter/sort the Jobs Completed page based on the job statistics fields. I think this would be a great enhancement and would recommend you post an idea.

As a workaround, I'd recommend you use the REST API GET /transformations/jobs/completed to fetch this data and then build a workspace to create the dashboard you desire. Here is a tutorial with some examples: https://community.safe.com/s/article/Monitoring-FME-Server-Job-Activity-using-the-REST-API

 

I can certainly log an Idea.

 

Regarding REST API, this is a workable solution. I can build a tableau or powerBI dashboard using the rest API data or create a workbench (with help of our user) to publish the FME dashboard. Do we have any such advanced FME dashboards published by safe software for admin consumption already?

 

If I take the tableau or PowerBI path ..

What is the unit of memory in the output of /transformations/jobs/completed ? If I need to get MB will I divide it by 1024*x*8 or 1024*x? what will be x?

N
Badge

Hi @namanpatel​ ,

It is not currently possible to filter/sort the Jobs Completed page based on the job statistics fields. I think this would be a great enhancement and would recommend you post an idea.

As a workaround, I'd recommend you use the REST API GET /transformations/jobs/completed to fetch this data and then build a workspace to create the dashboard you desire. Here is a tutorial with some examples: https://community.safe.com/s/article/Monitoring-FME-Server-Job-Activity-using-the-REST-API

 

Idea Created: AC Idea: FME Server - Allow User to sort by any visible column or multiple columns (safe.com)

N
Badge

Hi @namanpatel​ ,

It is not currently possible to filter/sort the Jobs Completed page based on the job statistics fields. I think this would be a great enhancement and would recommend you post an idea.

As a workaround, I'd recommend you use the REST API GET /transformations/jobs/completed to fetch this data and then build a workspace to create the dashboard you desire. Here is a tutorial with some examples: https://community.safe.com/s/article/Monitoring-FME-Server-Job-Activity-using-the-REST-API

 

Another Idea: AC Idea: FME Server - Allow configurable Filter just like columns configuration (safe.com)

H
Badge +2

I can certainly log an Idea.

 

Regarding REST API, this is a workable solution. I can build a tableau or powerBI dashboard using the rest API data or create a workbench (with help of our user) to publish the FME dashboard. Do we have any such advanced FME dashboards published by safe software for admin consumption already?

 

If I take the tableau or PowerBI path ..

What is the unit of memory in the output of /transformations/jobs/completed ? If I need to get MB will I divide it by 1024*x*8 or 1024*x? what will be x?

Hi @namanpatel​ ,

No, unfortunately, we don't have examples of integrating the FME Server REST API with any BI tools.

 Great question, I'll make a request to include this information in our documentation. The REST API returns the Peak Memory Usage as Bytes, so to display this value as MB you'll need to divide by 1000000.

N
Badge

I can certainly log an Idea.

 

Regarding REST API, this is a workable solution. I can build a tableau or powerBI dashboard using the rest API data or create a workbench (with help of our user) to publish the FME dashboard. Do we have any such advanced FME dashboards published by safe software for admin consumption already?

 

If I take the tableau or PowerBI path ..

What is the unit of memory in the output of /transformations/jobs/completed ? If I need to get MB will I divide it by 1024*x*8 or 1024*x? what will be x?

My question about FME Dashboard was regarding like FME workspaces that uses FME API to produce the results I am looking for, do we have something which I can start off?

H
Badge +2

I can certainly log an Idea.

 

Regarding REST API, this is a workable solution. I can build a tableau or powerBI dashboard using the rest API data or create a workbench (with help of our user) to publish the FME dashboard. Do we have any such advanced FME dashboards published by safe software for admin consumption already?

 

If I take the tableau or PowerBI path ..

What is the unit of memory in the output of /transformations/jobs/completed ? If I need to get MB will I divide it by 1024*x*8 or 1024*x? what will be x?

Hi @namanpatel​ ,

The article I linked in my first response above is the only resource I am aware of that could help you get started. I created that article, so if you have any questions about the example workspaces please let me know and I can try my best to provide further direction.

N
Badge

I can certainly log an Idea.

 

Regarding REST API, this is a workable solution. I can build a tableau or powerBI dashboard using the rest API data or create a workbench (with help of our user) to publish the FME dashboard. Do we have any such advanced FME dashboards published by safe software for admin consumption already?

 

If I take the tableau or PowerBI path ..

What is the unit of memory in the output of /transformations/jobs/completed ? If I need to get MB will I divide it by 1024*x*8 or 1024*x? what will be x?

Sorry I missed that expand button on the first post. This will certainly be helpful, that's what I was looking for to get started.

N
Badge

Please see if you can advocate for these ideas internally within Safe.

N
Badge

I can certainly log an Idea.

 

Regarding REST API, this is a workable solution. I can build a tableau or powerBI dashboard using the rest API data or create a workbench (with help of our user) to publish the FME dashboard. Do we have any such advanced FME dashboards published by safe software for admin consumption already?

 

If I take the tableau or PowerBI path ..

What is the unit of memory in the output of /transformations/jobs/completed ? If I need to get MB will I divide it by 1024*x*8 or 1024*x? what will be x?

How does the PeakMemoryUsage_BoxPlot iterate through pages if there are more than 2000 records? there seems only 2 HTTPCaller and I am not sure if there are any recursive calls involved there.

N
Badge

I can certainly log an Idea.

 

Regarding REST API, this is a workable solution. I can build a tableau or powerBI dashboard using the rest API data or create a workbench (with help of our user) to publish the FME dashboard. Do we have any such advanced FME dashboards published by safe software for admin consumption already?

 

If I take the tableau or PowerBI path ..

What is the unit of memory in the output of /transformations/jobs/completed ? If I need to get MB will I divide it by 1024*x*8 or 1024*x? what will be x?

Ignore my previous question, I think I understand the recursion logic through the cloner transformer. still learning

H
Badge +2

I can certainly log an Idea.

 

Regarding REST API, this is a workable solution. I can build a tableau or powerBI dashboard using the rest API data or create a workbench (with help of our user) to publish the FME dashboard. Do we have any such advanced FME dashboards published by safe software for admin consumption already?

 

If I take the tableau or PowerBI path ..

What is the unit of memory in the output of /transformations/jobs/completed ? If I need to get MB will I divide it by 1024*x*8 or 1024*x? what will be x?

Edit: Sorry I wrote this out before I saw your second post - hopefully it is still helpful.

 

Hi @namanpatel​ ,

 

The FME Server API supports pagination using offset and limit parameters supplied in the query string:

 

limit - Number of results to return in the request

offset - Result number to start request from

 

The maximum limit allowed is 1000, meaning you can return a maximum of 1000 jobs in a single request. To fetch the following page you must then set the offset to 1000, so the next response contains 1000 jobs starting from job 1001 etc.

 

This is set up in the workspace in the transformers contained within the first bookmark named '1: Fetch Job Statistics with pagination':

 Screen Shot 2022-04-06 at 7.35.41 AM

  1. The first nested bookmark contains an HTTPCaller that fetches an attribute called 'totalCount' which is the total number of jobs.
  2. The second nested bookmark uses this value to create the offset. A new feature is created for each offset that is required. E.g. if you have 10100 jobs then you'll have offsets from 0 through to 10000.
  3. The third nested bookmark contains the HTTPCaller and this is the one fetching all the records. It uses the offset value created in the previous step to fetch jobs from every page.

 

Reply


Community Stats

28,709
Posts
109,481
Replies
36,554
Members
Terms & ConditionsCookie settings
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.