Dear FME friends,
Welcome to What's Up Wednesday.
For reasons of space, this is an extract from the full version of this newsletter, which you can find here on LinkedIn.
Streaming to a Database
This week I saw a question about writing an incoming stream of data to a database:
"I have a workspace that runs continually in streaming mode (receives from a connector). I want to write to a Spatialite DB while the workspace is still running. I've tried a FeatureWriter, but nothing is written out while the workspace is running. Is this normal behaviour for Workbench?"
So there is a great article on the knowledge base about how to write to a database during a streaming mode. Writing is a little tricky because, generally, writers don't do anything until they have all the incoming data available, yet that's not going to work in a continuous stream.
The article explains that you have to set Bulk Insert parameters, including the Features Per Transaction:
Here, the writer waits for 1,000 features before submitting them to the database. That may be fine, but if I'm only receiving 100 messages per hour, then it'll be 10 hours before anything gets written. So you have to know how much data you're expecting to set this to a suitable value.
But if you don't set anything, then the data will never get written, because the writer is always waiting for more.
Another method is to use the FeatureWriter in Group mode. This is especially useful in conjunction with the TimeWindower when you want to commit data on a time interval.
Here I write data at 5-minute intervals. It's important to set the group change to "When Group Changes", otherwise FME goes back to waiting for all of the data. Another interesting twist is that the data isn't written until a feature with a new time window arrives. If there is no new feature, the old data doesn't get written. So that's just something to be aware of.
But FME can certainly write to a database while reading data from a stream, as the above information shows.
Mapping Month and FME Webinars
I call November "mapping month" because it includes GIS Day, but mostly because it's also time for the #30DayMapChallenge. The idea is to create a map every day in November. You can find some excellent maps posted on Twitter, and on LinkedIn, like this one.
Are you taking part? Would you use FME if you do? I saw this week a great example of creating maps automatically using FME, in this video from GRAPH LAND.
But if you want to create more refined cartography, then sign up for our webinar entitled "Streamlining Processes Towards High-Quality Cartography".
I'll be presenting alongside partners Hans from Red Geographics and Nick from Avenza. Maps need data and we'll show how FME can help getting data in the right place in the right structure, so you can spend less time struggling with data and more time making creative maps.
I'll even show how I used FME to help make this map for the Map Projection Trading Cards project:
Other upcoming webinars are:
[Nov 23] Rapid Web Development with Server Apps in FME 2022.2
[Dec 7] The Secret to Mapnik Mastery with FME
[Dec 13] 12 Best Practices to Get the Most Value from the FME Platform
Conference Videos
If you didn't attend the FME User Conference earlier this year, you missed a whole bunch of amazing talks and presentations. You might also have missed that you can view most of these now by signing up for digital access on our website.
On the subject of conference presentations, Safe partner Avineon-Tensing is at the Esri European Energy Resources GIS Conference in London (November 7/8), where they're talking on the subject of "Pipeline Industry Analytics and Visualization".
They're also at the OGC Location Powers conference in London (November 15/16), where you should look out for account manager Oliver Morris, who is presenting on the topic of "mixing company fixed asset data with geospatial data".
Oliver also has an article online about the advantages of low-code/no-code tools - like FME - and why they are like the baking shows you see on TV!
Miscellaneous News
- The FME Hub monthly digest is now available for October.
- Check out my colleague Dmitri's tweet about using FME with an ML library to carry out tree detection on satellite imagery. Hopefully, he'll be able to turn it into a custom transformer as promised.
- We're in Palm Springs for the Esri Infrastructure Management & GIS Conference this week. Drop by stand 304 to say hello.
- Red Geographic's "Transformer Friday" this week is the NeighborFinder.
- From Poland, a case study from Safe partner Globema about working with telecommunications operator Nexera.
