Skip to main content
Question

FME "BatchMode" Missing vs Null - Performance

  • June 14, 2023
  • 4 replies
  • 31 views

virtualcitymatt
Celebrity
Forum|alt.badge.img+47

Is there a difference in performance when it comes to working with Missing vs Null values in FME?

 

I'm working with some database tables which have a lot of null values. The output is something like JSON so In the end I want the null values to be missing in the output rather than a null or an empty string.

 

but when it comes for manipulating the data is there a difference between missing and null values?

4 replies

hkingsbury
Celebrity
Forum|alt.badge.img+63
  • Celebrity
  • 1620 replies
  • June 15, 2023

Not a definative answer, but my thought process would be that missing is faster as at the feature schema level, it doesn't exist, fewer fields, less data, faster processing....

 

But I might be wrong!


virtualcitymatt
Celebrity
Forum|alt.badge.img+47
  • Author
  • Celebrity
  • 2000 replies
  • June 21, 2023

Not a definative answer, but my thought process would be that missing is faster as at the feature schema level, it doesn't exist, fewer fields, less data, faster processing....

 

But I might be wrong!

Yeah thanks Hamish - My thinking was that with Bulk Mode it's better to have all the features share the same table structure. I'm not sure if having a lot of missing attributes messes that up. Any ideas @mark2atsafe​ ?

I did some minor testing but results were inconclusive


mark2atsafe
Safer
Forum|alt.badge.img+56
  • Safer
  • 2554 replies
  • June 22, 2023

Yeah thanks Hamish - My thinking was that with Bulk Mode it's better to have all the features share the same table structure. I'm not sure if having a lot of missing attributes messes that up. Any ideas @mark2atsafe​ ?

I did some minor testing but results were inconclusive

I asked around, and we don't really think there would be a huge difference. You'd need to have hundreds of null/empty attributes per feature to even make it worth testing. But if I hear different from another source, I'll let you know.


virtualcitymatt
Celebrity
Forum|alt.badge.img+47
  • Author
  • Celebrity
  • 2000 replies
  • June 23, 2023

Yeah thanks Hamish - My thinking was that with Bulk Mode it's better to have all the features share the same table structure. I'm not sure if having a lot of missing attributes messes that up. Any ideas @mark2atsafe​ ?

I did some minor testing but results were inconclusive

Awesome thanks mark. definitely not hundreds - more in the dozens. Seems to be working well either way so I'm happy.