Skip to main content

Hi,
I’ve been working on a pretty large export workbench.

The subject is related to real estate. We’ve got a relational database and we drill into about 20 external layers to get some info and to filter our own data.

The workbench has grown a lot over time and now it’s really hard to maintain or yet expand. I’m afraid it might break anywhere.

Are there best practices on how to go about many processing jobs ?

Is it sometimes best to split a workbench into several separate workbenches ? If yes, how to structure the processing parts ?

Where to start the cleaning up chore ?

Are there any parts of the workspace that could be reusable and therefore candidates for a Custom Transfomrer?

 

Even if there isn’t any, I would create Custom Transformers to group functionality to make the main project easier to read and to mantain.


Hi Oscard,

Thank you.

More Custom Transformers sounds like a good idea.

However, there are many fields which get used in different parts of the workbench. Sometimes for an evolution I need to change the name of an attribute, for instance with BulkAttributeRenamer. Might it be more difficult to maintain for such cases if processes are isolated in CustomTransformers ?

Would you have any recommandations to manage large numbers of attributes ? Can one think of attributes a little like the concept of API ? At such a point, I should have these attributes and not more… I suppose that would be the AttributeKeeper.

Kind regards,

Timothée


My first comment would be to always back up and save what works to begin with before making any changes. Working in a new version helps should things go wrong.

The custom transformers is a great idea, i’m also a fan of bookmarks especially now that you can minimise a bookmark to only show the input and output transformers.

With regards to attributes, to help keep workbenches efficient, only reading required attributes and removing attributes which are not longer needed is always a good idea.

Generally, AttributeKeeper (keeping less than you’re removing) or AttributeRemover (removing less than you’re keeping). Otherwise AttributeManagers can also help you do more in a single transformer. There’s many ways to skin a cat on this one.


Reply