Use of Recorder / Player in comparison with FeatureWriter / FeatureReader

Hi @jeroenstiers, the regular FFS writer also has the Compression Level parameter and it can be set even if you use the writer with the FeatureWriter. Check the "Parameters" button after setting the Format to FFS.

In addition to what @david_r said the Recorder/Player can also be helpful if I'm troubleshooting an issue for a client and they have problems with data coming out of a database that I don't have access to unless I'm physically in their office. That way I can still see what the source data looks like and try and help them.

I have only recently discovered the Recorder and Player and am too not clear on when I should use these vs FFS writer & reader. I am also not understanding what the Recoder settings to store scanned schema and metadata do. With them enabled, I end up with a considerably larger file but nothing changes when I use the Player to read them back into a workspace: the schema is still not there (<no schema> column only) However if I read with the FFS reader, the schema comes in whether or not I enabled the setting in the Recorder! (I don't have to expose anything for the columns to display). Quite confusing.

The use-case for me is that I have a workspace we run internally that generates 8k to 9k records. I need to make an API call for 5k of those records and it takes over 6 hours to finish so I cannot run internally on FME server and lock everything up. My plan is to use the Revit Server REST API and write it to a resource location on FME Cloud, then have a directory watcher kick off a workspace to read this data and process there when the server is not busy and spin up more engines as needed.

The Recorder/Player are the slightly rule breaking country cousins of the more refined FFS Reader/Writer. They play a bit fast and loose with the rules. If you ask the Recorder to make a schema and store metadata, you'll get an FFS file that should display more quickly in the data inspector. It will also read a bit quicker with the FFS reader, since the schema is known.

The Player is a form of FFS reader that doesn't care a hoot about schema. You're on your own to manually expose (AttributeExposer) what was in there. But today with Feature Caching it's not as hard as it use to be.

All things equal, I think in your situation I'd stick with the FFS Reader/Writer (in FeatureReader/FeatureWriter if you need them mid stream). I think the results will be a bit more predictable.

Also will throw in that we've recently been doing tests with modern compression algorithms and getting some "tastes great AND less filling" type results (i.e smaller files and faster to read and write). So some improvements coming to FFS land in the near future too.

Thanks a lot @Dale Lutz​ for that detail and recommendations! I have managed to get the above done now using a Recorder to write the .FFS file (small file size, straightforward) and then both a FeatureReader and regular Reader to read back them in.

In the above case, I'm also using the FME Server REST API to write a password-protected FFS from our internal network to a resource location on FME Cloud, then an automation with a directory watcher kicks in to run a workspace, read the written FFS and continue from there. The latter also writes a new FFS locally on FME Cloud if it cannot process all the records so it picks up from there the next time it runs and so on, effectively cycling through a large dataset that is "chained in a circle" (end to start) and only executing a chunk at a time based on a limit of records we can execute daily. It is a file format that I will be using quite frequently it seems!