Question

Best Data Capture Practices?

C

Hi guys,

I recently joined a company and I look after a central system. My system integrates with many other systems to bring in data and host it in this central system.

The big issue I found is the way various teams are capturing the data in their specific area and then struggling to deliver the data to my central system.

The way they capture the data is through using excel forms or other mechanisms that allow for data entry mistakes. Also, the processes to push into my system is manual from their end.

I was hoping to ask on here if you can assist with best practices to help develop and what experiences you had to improve this process:

  1. A quick, easy, and sufficient way for data capture to happen by the user who collects the data. Should we look into form entries where it can push into a database or other possibilities? What is the best way to do this?

  2. How to monitor data capture and processes (monitoring to see if users have for example submitted data or if they are late). If for example, they have not submitted data for say this week, then we can send an automated email to suggest they haven't

 

 

3 replies

deanhowell
Badge +9

Hello @cardiofaz, FME Server has lots of monitoring options that will allow work out when files / folders / databases are changed. You can use them to do a last edited check and then send a prompt out.

dparsons42

If your people are mobile and collecting data with a location component, check out "ESRI ArcGIS Online" AGOL, and "ESRI Collector" both will work great with FME once your data is collected and you can view the data on maps etc You can build an APP to collect the right data right from the spreadsheet you already have. You can then use FME to pull that data in house and process that data any way you want. Just one idea ..

H
Badge +2

Hi @cardiofaz,

 

Do you have FME Server at your organisation, I think this would help greatly with application integration here.

  1. For user collection data pushed to a db two options come to mind, however for automation FMEServer will be required.

     

     

    1. A form that can be connected to a webhook, meaning that the data is sent to the webhook on form submission. FME Server supports Webhooks as a Trigger, meaning that in can receive incoming form data in realtime which can then be processed in a job.

       

      We have an example of connecting Survey123 to FME using a webhook here. I know Google Sheets supports connection via a Webhook so Google Forms could be another option for you.

       

       

    2. An FME Server App, this is very similar to above, but rather than having a form created in a separate application that needs to talk to FME, the FME Server App is just hosting the workspace so on form submission the workspace is run directly. The front end for the user is a list of Published Parameters so it appears like any other form for them to fill in, and there are various types such a choice, plain text so you have some control with the form to help try and limit mistakes or inconsistencies. You can read more about FME Server Apps here.

       

       

      In both cases since the form submission is running a workspace, here the data can be checked and validated - if incorrect you could notify the user of the issues via email and ask them to resubmit and then if correct it can be written to the database.

       

       

  2. For monitoring data you'll likely want to set up a schedule to a workspace that reviews your database and performs some kind of test to determine if follow up emails need to be sent.

     

     

    For example when the user submits a form perhaps there is a field for their email. Separately you have a list of all emails you expect to receive a form from. You can perform a join on the two lists, and any that are unjoined you can then send out reminder emails to.

     

     

    In built Scheduling is supported in FME Server, or if you are only using FME Desktop this could be set up via Windows Task Scheduler.

     

    Sending an Email is also possible either via Automations in FME Server or the Emailer transformer in Desktop.

 

 

 

Reply


Community Stats

28,711
Posts
109,487
Replies
36,559
Members
Terms & ConditionsCookie settings
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.