Skip to main content
Solved

How to see which features have invalid source datasets when using a FeatureWrite?


simonlr
Contributor
Forum|alt.badge.img+2

Hi, i am copying files using the FeatureWriter with FileCopy and but some of them are missing but i dont get an attribute telling me which files are missing. All i get is this in the log:
File Copy Writer: Skipped 401 features that contained invalid source datasets
But what i would like is to route those with invalid source data set and have them write to an excel ark or similar so i know which files cant be found.
But right now the features all come out of the same port with no distinction of wheter or not the copy was actually succesful. Which is not very useful for me.
 

Any help is very much appriciated :)

Best answer by nielsgerrits

Ah correct, sorry my bad. Then use good old Python. A PythonCaller with the following code works.

import fme
import fmeobjects
import os

class FeatureProcessor(object):

    def __init__(self):
        pass
  
    def input(self, feature: fmeobjects.FMEFeature):
        if os.path.exists(feature.getAttribute('FilePath')):
            feature.setAttribute("FileCheck","true")
        else:
            feature.setAttribute("FileCheck","false")
        self.pyoutput(feature)

    def close(self):
        pass

View original
Did this help you find an answer to your question?

6 replies

nielsgerrits
VIP
Forum|alt.badge.img+56

One way to solve this is to use the Directory and File Pathnames reader to confirm if the file does exist or not, and only send the confirmed file features to the File Copy writer.


simonlr
Contributor
Forum|alt.badge.img+2
  • Author
  • Contributor
  • July 15, 2025

 

nielsgerrits wrote:

One way to solve this is to use the Directory and File Pathnames reader to confirm if the file does exist or not, and only send the confirmed file features to the File Copy writer.

Thank you for your answer.

How i wish this worked, that you could just set the source file as the data set for Directory and File Pathnames, but i seems it only reads folders not individual files.
Is there a way to do this without having a Directory and File Pathnames reader read everything? It takes a very very long time.
I have a 4TB harddrive with millions of files and i really only need to copy a few specific files over every time, but i do need to know if some files were not succesfully copied and which files they are.


nielsgerrits
VIP
Forum|alt.badge.img+56
  • Best Answer
  • July 15, 2025

Ah correct, sorry my bad. Then use good old Python. A PythonCaller with the following code works.

import fme
import fmeobjects
import os

class FeatureProcessor(object):

    def __init__(self):
        pass
  
    def input(self, feature: fmeobjects.FMEFeature):
        if os.path.exists(feature.getAttribute('FilePath')):
            feature.setAttribute("FileCheck","true")
        else:
            feature.setAttribute("FileCheck","false")
        self.pyoutput(feature)

    def close(self):
        pass


simonlr
Contributor
Forum|alt.badge.img+2
  • Author
  • Contributor
  • July 15, 2025

Thank you very much :)
It works great. I am going to save that as a custom transformer to use again another time.


Have a great day.


nielsgerrits
VIP
Forum|alt.badge.img+56

Cheers :)


takashi
Evangelist
  • July 15, 2025

Hi ​@simonlr .

FileExistenceChecker from FME Hub could also be helpful ;-)


Cookie policy

We use cookies to enhance and personalize your experience. If you accept you agree to our full cookie policy. Learn more about our cookies.

 
Cookie settings