Solved

Use of Recorder / Player in comparison with FeatureWriter / FeatureReader

  • 20 January 2017
  • 8 replies
  • 1 view
J
Badge +7

This morning I stumbled upon the Recorder and Player. After reading the documentation I was really eager to try them out in a workspace for storing temporary files. We also tested the performance by comparing with a FeatureReader / FeatureWriter and we didn't find any big differences when the compression-level of the Recorder was low.

However when increasing the compression level, the performance was increasingly worse with the Recorder in comparison with the FeatureWriter.

 

Is the only advantage of the Recorder in stead of the FeatureWriter that it can compress the data?

 

When do you use the one or the other?

Thanks for your thoughts!

icon

Best answer by david_r 20 January 2017, 11:00

View original

8 replies

david_r
Rank
Userlevel 4

The Recorder / Player combo has been there for a long time (long before the FeatureWriter) and I've used them quite a bit for debugging purposes. Notably, they are very handy if you're struggling with an issue in the middle of a very time consuming translation and you want to isolate a certain behavior without having to run the entire workspace every time. Another example, let's say your workspace reads from a webservice or a very slow database. In that case you can connect a Recorder straight after the readers to quickly make a local copy of the dataset and then using the Player for subsequent testing, removing them before going into production. Finally, they can be very handy when communicating support issues to Safe as you can remove external dependencies in your workspace by supplying ffs files from the Recorder.

Regarding compression, that is always something that adds a lot of overhead, as it is an extra step that's quite heavy on math. With today's cheap disk space I hardly ever use compression, it simply isn't worth it, I find, unless I intend to send data to someone and the network performance is noticably worse than the time it takes to compress/decompress the dataset.

As for how I see it:

  • FeatureWriter is to write the results of your workspace
  • Recorder / Player is for debugging purposes

takashi
Rank
Userlevel 2
Badge +17

Hi @jeroenstiers, the regular FFS writer also has the Compression Level parameter and it can be set even if you use the writer with the FeatureWriter. Check the "Parameters" button after setting the Format to FFS.

J
Badge +7

The Recorder / Player combo has been there for a long time (long before the FeatureWriter) and I've used them quite a bit for debugging purposes. Notably, they are very handy if you're struggling with an issue in the middle of a very time consuming translation and you want to isolate a certain behavior without having to run the entire workspace every time. Another example, let's say your workspace reads from a webservice or a very slow database. In that case you can connect a Recorder straight after the readers to quickly make a local copy of the dataset and then using the Player for subsequent testing, removing them before going into production. Finally, they can be very handy when communicating support issues to Safe as you can remove external dependencies in your workspace by supplying ffs files from the Recorder.

Regarding compression, that is always something that adds a lot of overhead, as it is an extra step that's quite heavy on math. With today's cheap disk space I hardly ever use compression, it simply isn't worth it, I find, unless I intend to send data to someone and the network performance is noticably worse than the time it takes to compress/decompress the dataset.

As for how I see it:

  • FeatureWriter is to write the results of your workspace
  • Recorder / Player is for debugging purposes

Thanks @david_r

 

 

I like your suggestion. By using the transformers in specific situations (debugging / standard writing) it will allow other FME developers to quickly see what you are doing.

 

Thanks!

 

redgeographics
Rank
Userlevel 5
Badge +25

In addition to what @david_r said the Recorder/Player can also be helpful if I'm troubleshooting an issue for a client and they have problems with data coming out of a database that I don't have access to unless I'm physically in their office. That way I can still see what the source data looks like and try and help them.

fmelizard
Rank
Userlevel 4
Badge +13

The Recorder / Player combo has been there for a long time (long before the FeatureWriter) and I've used them quite a bit for debugging purposes. Notably, they are very handy if you're struggling with an issue in the middle of a very time consuming translation and you want to isolate a certain behavior without having to run the entire workspace every time. Another example, let's say your workspace reads from a webservice or a very slow database. In that case you can connect a Recorder straight after the readers to quickly make a local copy of the dataset and then using the Player for subsequent testing, removing them before going into production. Finally, they can be very handy when communicating support issues to Safe as you can remove external dependencies in your workspace by supplying ffs files from the Recorder.

Regarding compression, that is always something that adds a lot of overhead, as it is an extra step that's quite heavy on math. With today's cheap disk space I hardly ever use compression, it simply isn't worth it, I find, unless I intend to send data to someone and the network performance is noticably worse than the time it takes to compress/decompress the dataset.

As for how I see it:

  • FeatureWriter is to write the results of your workspace
  • Recorder / Player is for debugging purposes

Agree completely with David's analysis. Ironically we were talking yesterday at lunch about how that compression argument is not so interesting any longer. 15 years ago disks were so much slower than CPU so we thought it would be worth squeezing the data to save time. Not so any longer. In fact, we were thinking about taking that setting away at some point.

 

 

dbaldacchino1
Rank
Badge +3

I have only recently discovered the Recorder and Player and am too not clear on when I should use these vs FFS writer & reader. I am also not understanding what the Recoder settings to store scanned schema and metadata do. With them enabled, I end up with a considerably larger file but nothing changes when I use the Player to read them back into a workspace: the schema is still not there (<no schema> column only) However if I read with the FFS reader, the schema comes in whether or not I enabled the setting in the Recorder! (I don't have to expose anything for the columns to display). Quite confusing.

 

The use-case for me is that I have a workspace we run internally that generates 8k to 9k records. I need to make an API call for 5k of those records and it takes over 6 hours to finish so I cannot run internally on FME server and lock everything up. My plan is to use the Revit Server REST API and write it to a resource location on FME Cloud, then have a directory watcher kick off a workspace to read this data and process there when the server is not busy and spin up more engines as needed.

fmelizard
Rank
Userlevel 4
Badge +13

The Recorder/Player are the slightly rule breaking country cousins of the more refined FFS Reader/Writer. They play a bit fast and loose with the rules. If you ask the Recorder to make a schema and store metadata, you'll get an FFS file that should display more quickly in the data inspector. It will also read a bit quicker with the FFS reader, since the schema is known.

 

The Player is a form of FFS reader that doesn't care a hoot about schema. You're on your own to manually expose (AttributeExposer) what was in there. But today with Feature Caching it's not as hard as it use to be.

 

All things equal, I think in your situation I'd stick with the FFS Reader/Writer (in FeatureReader/FeatureWriter if you need them mid stream). I think the results will be a bit more predictable.

 

Also will throw in that we've recently been doing tests with modern compression algorithms and getting some "tastes great AND less filling" type results (i.e smaller files and faster to read and write). So some improvements coming to FFS land in the near future too.

dbaldacchino1
Rank
Badge +3

Thanks a lot @Dale Lutz​ for that detail and recommendations! I have managed to get the above done now using a Recorder to write the .FFS file (small file size, straightforward) and then both a FeatureReader and regular Reader to read back them in.

 

In the above case, I'm also using the FME Server REST API to write a password-protected FFS from our internal network to a resource location on FME Cloud, then an automation with a directory watcher kicks in to run a workspace, read the written FFS and continue from there. The latter also writes a new FFS locally on FME Cloud if it cannot process all the records so it picks up from there the next time it runs and so on, effectively cycling through a large dataset that is "chained in a circle" (end to start) and only executing a chunk at a time based on a limit of records we can execute daily. It is a file format that I will be using quite frequently it seems!

Reply


Community Stats

28,723
Posts
109,511
Replies
36,560
Members
Terms & ConditionsCookie settings
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.