Skip to main content
Question

Run tester on every date field in a table

  • February 19, 2025
  • 2 replies
  • 34 views

markcoopersdc
Contributor
Forum|alt.badge.img+6

I have a large number of spatial tables, each potentially having dozens of date fields. I need to check the dates in each of these fields to check they are greater than a certain data (01/01/1900?). These tables are being taken from Oracle and uploaded to AGOL. If there are any dodgy dates, it appears to break AGOL and no spatial data appears for the whole dataset!

Is there a way to automatically select fields with a type of date or date time and check validate each row for each date field is above that date? That way I could investigate and correct before the data is transferred. There doesnt seem to be any kind of filter for attribute type.

Once I have done this, I will need to do something similar with varchar fields as AGOL doesnt like certain characters.

Many thanks

 

 

2 replies

liamfez
Influencer
Forum|alt.badge.img+34
  • Influencer
  • February 19, 2025

There is a “Type Is” operator available in the test conditions. So you could use a Tester or TestFilter and  select an attribute to test if the values for that attribute are of a specific type like datetime.

You could try working with the AttributeExploder prior to the Tester/TestFilter with datetime filtering to find attributes that have datetime values.


bwn
Evangelist
Forum|alt.badge.img+26
  • Evangelist
  • February 20, 2025

The list of date fields can be determined fairly easily by a FeatureReader in “Schema Only” mode.   Each Schema Feature output (1 per table) will have a List of all Fields and their Data Type.  These can be filtered by ListExploder then Tester.

 

To test this list of derived date/time attributes for date values dynamically is quite a bit trickier.  I’ve trialled a few approaches, but one example approach is:

  1. Use the results from the FeatureReader ListExploder Tester of Feature Type Names vs Date/Time Field Names to use for build Oracle SELECT SQL strings that only select these fields per Table
  2. Use an AttributeCreator to add a pair of “” around each field name output by the Tester
  3. Then use an Aggregator to to build a comma-delimited list of field names to SELECT (Aggregator Group By Parameter = Feature Type Name)
  4. Then use SQLExecutor to “SELECT @Value(<Aggregator Comma-delimited Field List>) FROM @Value(<Feature Type Name>)”
  5. Use the SQLExecutor results in AttributeExploder → Tester (or AttributeExploderStatisticsCalculator) to flag fields or rows that have values that are “out of bounds”

Cookie policy

We use cookies to enhance and personalize your experience. If you accept you agree to our full cookie policy. Learn more about our cookies.

 
Cookie settings