Automation Efficiency

First tip in FME form is to turn off the feature cache

If you have FME Flow - consider creating an automation

If the processing must remain separate, add an Automation Writer from FME Form.

The workspace can batch features and hand them off to an Automation rather than creating a new Flow job for every record.

This allows you to:

• Control throughput.
• Batch multiple features into a single message.
• Reduce queue pressure dramatically.
• Maintain decoupled processing.

If you only have Form, then batch the records with a parent workspace creating groups, then using a workspace runner, pass the batch identifier or SQL to the second workspace, where you only run X records per run instead of 300k

Why would you want to do it that way? Why not use an SQLCreator (or if the process is triggered by a single feature/event/message, an SQLExecutor) to read the rows from the database, then do whatever it is needs doing with those rows in a single workspace? It’s one of the reasons I haven’t been using Automations yet, there really isn’t much reason for me to do so. 

Make sure to change the defaults on the SQLCreator as well, so it reads those 300.000 rows in as few batches as possible. If you really need to process all of those rows one by one, take a look to see if you can do the work in the database itself with some clever sql. For all modern RDBMS’ses 300.000 rows is peanuts, so I always try to let them do as much work as possible.

definitely a database driven workspace has many advantages, I prefer orchestrating sub jobs/routines myself

either way you can’t get away from FME being a feature/record based processer. Performance can only be improved

• turning off the cache in Form(use inspector transformer where you need a result to inspect
• with cache, collapse bookmarks 
• minimising group based transformers
• jdbc over odbc?
• using SQL to run before or after writing - advanced pattern - disable and enable index your table. Without indexes:
  • Faster bulk inserts
  • Reduced transaction logging
  • Less page splitting

Working with Databases & Data Warehouses and FME – FME Support Center

• Database Tips & Tricks In FME

Whilst 300,000 features seems a lot, it should be managable by FME in one process. The (simplified) way I read your problem is you want to find the closest set of features to your input lines.

The NeighbourFinder should be able to handle this amount of features pretty easily