How to convert this to run in FME to pass all excel file types .xlsx out to an attribute. What I want to achieve is to recurse into every folder on my list, recursing into subfolders to identify all .xlsx files and return their file paths.

A Directory and File Pathnames reader would be the FME way to achieve this

Basic Single Implementation

You can use a path reader to read in a single folder location:

And then just set the text search for excels (*.xlsx) and say yes to look in subfolders:

When setup you will get attributes for all the file paths and other ancillary information as attributes. This would satisfy looking into 1 folder location with subfolders (if present).

For a recursive approach using your csv:

You could read in the csv and whatever attribute contains the folder directories you need to search (with subfolders) for excels you could link that to a FeatureReader transformer and then populate the settings as above as a PATH reader. To make it dynamic you would just use your csv attributes to populate the base folder path in the FeatureReader settings.

So each "feature" from the csv will enter the FeatureReader and populate the base folder directory where it should look and then find all the .xlsx's.

@fme​ you've said, twice, that "it fails". Can you elaborate? Are you getting an error message and if so, which one? How exactly have you set up your workspace?

@Hans van der Maarel​ i've resolved the error. However, after running i don't see port outputs like path_windows, path_unix, etc. (the output ports seen from using the standalone directory and path reader). Do i need to configure something else on the featurereader to see this?

@Hans van der Maarel​ i've resolved the error. However, after running i don't see port outputs like path_windows, path_unix, etc. (the output ports seen from using the standalone directory and path reader). Do i need to configure something else on the featurereader to see this?

@Hans van der Maarel​ i've resolved the error. However, after running i don't see port outputs like path_windows, path_unix, etc. (the output ports seen from using the standalone directory and path reader). Do i need to configure something else on the featurereader to see this?

@Hans van der Maarel​ i've resolved the error. However, after running i don't see port outputs like path_windows, path_unix, etc. (the output ports seen from using the standalone directory and path reader). Do i need to configure something else on the featurereader to see this?

@Hans van der Maarel​ i've resolved the error. However, after running i don't see port outputs like path_windows, path_unix, etc. (the output ports seen from using the standalone directory and path reader). Do i need to configure something else on the featurereader to see this?

Is there a way to implement batch processing in fme workbench? The path reader is currently taking days to go through the folders to retrieve the files. Is there a varied efficient way to do this?

I've seen reports previously of poor performance with the path reader when recurse is set to yes

Perhaps you could see if a python solution is any quicker?

import fme
import fmeobjects
import glob
 
class FeatureProcessor(object):
    
    def __init__(self):
        pass
 
    def input(self, feature):
        root_dir = feature.getAttribute('directory')
        for filename in glob.iglob(root_dir + '/**/*.xlsx', recursive=True):
            feature.setAttribute("filename",filename)
            self.pyoutput(feature)
 
    def close(self):
        pass   

I've seen reports previously of poor performance with the path reader when recurse is set to yes

Perhaps you could see if a python solution is any quicker?

import fme
import fmeobjects
import glob
 
class FeatureProcessor(object):
    
    def __init__(self):
        pass
 
    def input(self, feature):
        root_dir = feature.getAttribute('directory')
        for filename in glob.iglob(root_dir + '/**/*.xlsx', recursive=True):
            feature.setAttribute("filename",filename)
            self.pyoutput(feature)
 
    def close(self):
        pass   

I've seen reports previously of poor performance with the path reader when recurse is set to yes

Perhaps you could see if a python solution is any quicker?

import fme
import fmeobjects
import glob
 
class FeatureProcessor(object):
    
    def __init__(self):
        pass
 
    def input(self, feature):
        root_dir = feature.getAttribute('directory')
        for filename in glob.iglob(root_dir + '/**/*.xlsx', recursive=True):
            feature.setAttribute("filename",filename)
            self.pyoutput(feature)
 
    def close(self):
        pass