Skip to main content
Solved

Querying log folders - writing all logs to one folder

sam_appleton
Contributor
Forum|alt.badge.img+7

Hi,

I'm trying to ingest log files into Azure monitor so I can send out organisation wide notifications but i've run into an issue.

 

When I look into the machine that flow is installed on, I can see that it is storing logs in folders in batches of 1000 and it will create new folder each time it hits a multiple of 1000. I can process files into Azure, but this doesn't seem to work with a wildcard in the /current/ folder - it needs a folder that already exists.

 

Is there a way to write these logs all to the same folder?

The file pattern/query i'm using:

 C:\\ProgramData......logs\\engine\\current\\jobs\\*\\job_*.txt

Many thanks,

 

Sam

Best answer by david_r

sam_appleton wrote:

Thank you for the response, David. Unfortunately the cloud team has dictated that this all must go through Azure. I'll try the double asterisk and see if that helps at all.

If you're using software supplied by Azure to ingest the log files you may have better luck asking on an Azure specific forum. I don't think it's possible or even desirable to have FME keep all the (potentially tens of thousands) job logs in a single directory.

View original
Did this help you find an answer to your question?

3 replies

david_r
Evangelist
  • david_rEvangelist
  • January 5, 2024

If you're using FME to iterate over the log files, you can use the following pattern to look for files in all sub-directories:

 C:\ProgramData......logs\engine\current\jobs\**\job_*.txt

Note the double asterisk to match any directory.

sam_appleton
Contributor
Forum|alt.badge.img+7
  • sam_appletonContributor
  • Author
  • Contributor
  • January 5, 2024
david_r wrote:

If you're using FME to iterate over the log files, you can use the following pattern to look for files in all sub-directories:

 C:\ProgramData......logs\engine\current\jobs\**\job_*.txt

Note the double asterisk to match any directory.

Thank you for the response, David. Unfortunately the cloud team has dictated that this all must go through Azure. I'll try the double asterisk and see if that helps at all.

david_r
Evangelist
  • david_rEvangelist
  • Best Answer
  • January 5, 2024
sam_appleton wrote:

Thank you for the response, David. Unfortunately the cloud team has dictated that this all must go through Azure. I'll try the double asterisk and see if that helps at all.

If you're using software supplied by Azure to ingest the log files you may have better luck asking on an Azure specific forum. I don't think it's possible or even desirable to have FME keep all the (potentially tens of thousands) job logs in a single directory.

Reply


Helpful Members This Week

Community Stats

31,092
Posts
117,882
Replies
38,849
Members

Latest FME

Terms & ConditionsCookie settings

Cookie policy

We use cookies to enhance and personalize your experience. If you accept you agree to our full cookie policy. Learn more about our cookies.

 
Cookie settings
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.