Skip to main content
  • EN

Transformers

Using transformers to modify data.

  • 13,027 Posts
  • 52,072 Replies
10000 Posts
lukeh81
Participant
lukeh81Participant
asked in Transformers
Multi CSV w/ Dual Reader Merging Help

Hi everyone,I have struck a roadblock in the creation of a particular FME Workbench that I now require help with. I appreciate your time in advance.I have multiple csv files that contains a table of data that I would like to process into a single csv file table. Lines 1-26 of these files contains general informationLine 27 contains headers for a table belowLine 28 onwards contains the data for the table. In Line 2 Row 2 there is a “cell” (B2) which contains text which I would like to add into a new column in the table repeatedly.I have done and been able to do the following:- Used 2 x CSV Readers- 1st CSV Reader data is transformed down to just cell B1 and B2, with B1 text being replaced with a header name 'Sensor_ID' and B2 simply being the original data in B2 eg. SN_ABC123- 2nd CSV Reader data is transformed to Line 27 onwards (i.e. just showing the table), plus a few other modifications using AttributeRenamer, AttributeRemover etc.- Tried using both FeatureMerger and FeatureJoiner t

484
lukeh81
Participant
vincenthischie1
Contributor
vincenthischie1Contributor
asked in Transformers
Distance Accumulation Alternative

Hello FME community, I would like to identify all locations that are within a 300m-perimeter around a polygon, taking into account barriers (large roads, rivers etc.). In ArcGIS there is a tool called “Distance Accumulation” that does just this, but I don’t want to depend on it. Thanks!

979
redgeographics
Celebrity
joy
Enthusiast
joyEnthusiast
asked in Transformers
How to remove rows with only NULL Values

Hello everyone, This might be a simple question, but for some reason I can't figure out how to remove rows with only Null Values (Of course I can useTestFiler or Tester, but would be cumbersome work, since I have a lot of columns and I don't want to go into that rabbit hole)These rows with Null values came into existence, since I read in an excel file with two worksheets with two different schemas. So the column names in worksheet 1, is different compared to the column names in worksheet 2. Therefore all the values from worksheet 2 have null values. I actually don't need worksheet 2, so I was thinking there might be 2 solutions. 1) don't read in worksheet 2 or2) remove the null values For both of them I don't know the answer.....dramatic sound effects..So if someone knows the answer, that would be very nice.

8088
woelfl
Participant
aashnaparikh
Contributor
aashnaparikhContributor
posted in Transformers
Attribute Conditioning

I have an excel file where i have 2 limits stated under single column and I want to split them as per table B where the Type = CLimit Type $30,000 per claim / $50,000 aggregate N $50,000 per claim / $70,000 aggregate C  Table BLimit Type Per claim Limit Aggregate Limit $50,000 per claim / $70,000 aggregate C $50,000 $70,000  

202
raghavendrans
Enthusiast
agelfert
Contributor
agelfertContributor
asked in Transformers
VertexExtractor not working in FME 2024.2

As mentioned in another post, I'm currently upgrading a bunch of workbenches to run on FME Flow 2024, some of them many years old. Typically, for anupgrade, I will go through each workbench, upgrade all transformers, readers, and writers, then run it in test desktop (workbench) environment, then publish to and test in a test server (flow) environment.In, FME 2024 the Vertex Extractor gives me bogus results, even after upgrading all the constituent transformers in this custom transformer. It looks like X and Y, or Lat and Lon, are reversed.When I open the CoordinateConcatenator (arrow above), I see this:Default settingWhile I haven’t dug through the rest of the custom transformer, this - frankly - looks counter-intuitive.When I change that to “XYZ”, everything runs just fine and results are as expected. My tweakI have tried to compare my version (I’m assuming that upgrading all components makes it uptodate) to older versions to see whether upgrading might have introduced this problem bu

14412
saraatsafe
Safer
aron
Supporter
aronSupporter
asked in Transformers
Identify an island in a lake on an island in a lake

I hope I got the title prepositions correct…I want to break apart lake data in order to generalize each category individually. The DonutHoleExtractor will “only” break apart “one level”. That is, independent-lakes and island-in-lakes-on-islands get sent one way and islands-in-independent-lakes and lakes-on-independent-islands get sent the other. Last option is to use an AreaOnAreaOverlayer or similar to sort everything correctly. But I would rather save that computing time since this is a huge dataset. Unfortunately the DonutHoleExtractor does not have an option for lists, or that would have been my chosen route, me thinks. Once generalized the geometries will be reassembled into donuts.  Q: Is there a way to deduct the relations between different types of polygons within the donuts without doing a load of spatial analysis? I would expect there to be some sort of relation info within the geometry type itself?edit: Rephrased the question for clarity  Pic for illustrative purposes 

835
aron
Supporter
cory
Contributor
coryContributor
asked in Transformers
When will the FME Projections for GDA2020 be fixed in FME Form and FME Flow?

Hi Team,I was wondering if there was an ETA as to when a permanent solution for fixing FME Form (and Flow) will be available, as we're still experiencing issues with reprojecting data in GDA 2020. The work around at the moment that we have been given is;- Open up the following file in Notepad (open as administrator) :"C:\Program Files\FME\Reproject\CsmapTransformationExceptions\TransformationExceptions.db" - Find the Australia section and add the following line : GDA2020-7P|GDA2020|XFORM|FWD|NULL_FME - Save!Cheers 

786
df_cos
Contributor
johnk
Contributor
johnkContributor
asked in Transformers
Looping with a FeatureReader

Hi FME world,I have two tables in a database (ArcGIS SDE). They are effectively a 1:1 match, linked via a common ID. Table 1 features are always present in the flow (filtered prior) and I want to read in ONLY the features from table 2, where match with the Table 1 filtered features (because there are 1m+).I’ve tried options like aggregating the Table 1 features and concatenating the ID, passing into Table 2 feature reader with a where clause. However, there is a maximum limit of queries the database can do (somewhere between 10,000-15,000 - and often I will need to read in more than this.My next thought is to create a looping transformer where I sample the T1 features into groups of 10k, aggregate, pass to FeatureReader, reads in associated 10k T2 features, then loop back to read in the next 10k, until there are no more features to be read (using a NoFeaturesTester to exit the loop.I’m a bit stuck on how to actually implement this though. Any ideas/thoughts much appreciated! Or if i’m

334
hkingsbury
Celebrity
tbsm223
Contributor
tbsm223Contributor
asked in Transformers
Question about HTTPCaller

Hey,  I have a question about the HTTP Caller. Is there a way to have the workspace it is in still continue even if one of the records fails to upload? The reason I ask is because there are some interesting errors I have ran into when uploading records to Cityworks, where something fails and then I have to re-run the entire workspace, after first deleting all of the inspections/work orders that were created.  The error involves an encoded string not being compatible with FME. I would just like a way to run the entire workspace for all features and not have it cancel out on a failure. Any tips?

151
hkingsbury
Celebrity
jpbarnes
Contributor
jpbarnesContributor
asked in Transformers
CKANConnector - Upload error HTTP 400: "Bad request - JSON Error: Invalid request. Please use POST method for your request

With the CKANConnector able to list Datasets and Resources without any issue, so I know CKAN Connection is good.   If I try to add and update a resource I receive the  “upload error HTTP 400: "Bad request - JSON Error: Invalid request. Please use POST method for your request”    I’m pulling the Dataset and Resource for the update resource from the dropdown.    Acts like a permission issue but I’m connecting as an Admin  on the openGov side. 

172
jpbarnes
Contributor
alexbiz
Supporter
alexbizSupporter
posted in Transformers
TimeSerieGenerator

Huge thanks to ​@mark2atsafe for granting me a shiny FME Star badge!It really gave me the push to finally share some of the custom tools I’ve built, starting with this one: the TimeSeriesGenerator. The idea came up when I needed to build a time‑series dataset to plot user activity over time. My end goal was to generate an HTML report showing how users interacted with an application. I was starting from a log file of user requests, but if no one made a request during a given period, I’d have gaps in my data and nothing to show in the report. This issue is a common one. Even FME Flow with his analytics page doesn’t handle time period with no data.This FME Flow is idling most of the time, but the Analytics page doesn’t report that.  That’s where the TimeSeriesGenerator comes in ! It creates a continuous series of timestamps between a start date and an end date, at a specified interval in seconds. You can then join your actual log data to those timestamps and be sure you’ll see a value (ev

430
alexbiz
Supporter
gennine
Contributor
gennineContributor
asked in Transformers
Conditional querry on an electric network

I'm working with an electrical network with feeders and branches. I'd like to add the coordinates of the intersection points with the feeders as attributes in the branches. I managed to do this in the first segment using the Intersector and SpatialFilter transformers, but I had trouble adding them to the downstream segments.I have a common field for each branch that can be obtained using networkTopology.So, my question is how to create conditional queries to add the coordinate values ​​of the intersection points with the downstream branches (see attached diagram).Thanks in advance. 

566
gennine
Contributor
mohamed.n
Contributor
mohamed.nContributor
asked in Transformers
Can't retrieve property sets data of dwg blocks

Hi, I'm trying to access the property sets within the Extended data from dwg file, the property sets are attached to block reference objects, quite similar to the case of this post.Attached below are two snapshots from how the property sets look like in Autocad and FME workbench featuring the two attributes that I was trying to expose with no successcan someone guide me through the detailed workflow, please? I'm using FME workbench 2024.2 Thanks,

435
raghavendrans
Enthusiast
gbcfgeo2025
Contributor
gbcfgeo2025Contributor
asked in Transformers
How to remove duplicates in Excel keeping only one record per ID

Hi everyone, quick question: how can I remove the rows from my Excel file where the ID number is duplicated, so that only one record per ID is kept?

669
veschwab
Contributor
ioannant
Contributor
ioannantContributor
asked in Transformers
renaming .dgn (Microstation) file levels with FME

HelloI am trying to figure out if it is possible to read Levels from a .dgn files, and rename them according to a Table given to the script. I work at a public contracting authority where we receive “As-Built” .dgn documentation from our consultants, structured according to industry standards regarding level naming, and it would be a huge help if we could rename them to a human readable name which our maintenance employees can understand.I guess I’ve found a way to “read” levels in Microstation files, but I cannot seem to find something that can write Levels. Is that possible with FME? Any help would be appreciated. CheersIoannis

353
raghavendrans
Enthusiast
geo-x
Contributor
geo-xContributor
asked in Transformers
Apply measure attributes to a line

Dear FME Community,I need to calculate measure PK from an object, modify them and apply to another object for cut it to elements.I try to explain all steps with this schema :I’m at the STEP 4 but I’m blocked when I want to apply my Measure PK to another objet.For that I’ve a FMWT in ZIP join to this subject.Thank you for your attention and your help.Geo-x

15815
geo-x
Contributor
xiaomengatsafe
Safer
xiaomengatsafeSafer
asked in Transformers
Join The FME Explorers Program - Help Shape FME's Future!

Introducing The FME Explorers ProgramJoin us in user experience studies to help explore new ideas and shape FME's future! I want to be an FME Explorer! How does it work?What am I signing up for?By signing up for the FME Explorers Program, you are letting us know that you are interested in participating in future user experience studies. We periodically conduct user experience studies to help us better understand how our users work with FME or data in general, and identify opportunities to make FME better.What to expect when I sign up?When you sign up, you'll be redirected to a short survey (won't take more than 5 minutes) to help us get to know you a little better. The information we gather from this survey will only be used to help us determine which of our future studies would be a good fit for you.When we find a user experience study that is a good match for you, we will email you an invitation. The invitation will include more details on that particular study, so you ca

2138
danilo_fme
Evangelist
danraleigh
Contributor
danraleighContributor
asked in Transformers
Creating Points Along a Polyline at Equal Intervals

I am hoping to produce a group of points along polylines of various lengths. I’ve been trying to mimic the GeneratePointsAlongLines() method from arcpy and ArcGIS Pro which will simply build out points from the start of a polyline at a designated interval (regardless of line segment vertices). I’ve investigated Densifier, but this transformer incorporates vertices and produces points that do not conform to the designated interval. I’ve contemplated utilizing Snipper, but this seems challenging given that many of the line segments I expect to incorporate may vary in lengths for different inputs.I’d appreciate any insight that can be offered.

1013
danraleigh
Contributor
rudewi
Participant
rudewiParticipant
asked in Transformers
Can I read the xdata of property sets of dwg blocks?

Hi. I'm reading dwg files with the civil 3D reader. I need to use the attributes in the propertysets. this works perfectly with exposing 'autocad_xrecord_datalist' in the reader for all geometry types, but not for the blocks. the property set attributes from the block don't seem to be present on the 'autocad_insert'. I tried exposing fme_attrib_info but they are also not there. any ideas? or is this just not possible on blocks ?i added a screenshot of the pset in the dwg

1375
mohamed.n
Contributor
lifalin2016
Contributor
lifalin2016Contributor
asked in Transformers
2024.2 yields an error in SharepointOnlineConnector ?

Hi.I built a workspace in 2024.1, that fetches a number of files from a customer Sharepoint site. It works (in 2024.1) without problems.I just installed 2024.2, and ran the workspace (with no changes).Now it issues an error with a SyntaxWarning from the package “safe.microsoft-sharepoint”:C:\Users\xxx\AppData\Roaming\Safe Software\FME\Packages\24783-win64\python\safe.microsoft-sharepoint\stringcase.py:247: SyntaxWarning: invalid escape sequence '\W' return re.sub("\W+", "", string)Is this something that I should worry about ?SharepointOnlineConnector is not listed as an upgradable transformer in 2024.2Cheers.

17818
sfb
Contributor
marta.podsiad
Supporter
marta.podsiadSupporter
asked in Transformers
Get Settings from AGOL layers

Hi,Does anyone know how to retrieve the information from the Settings tab of a layer published to Esri ArcGIS Online (AGOL)? I’m specifically trying to get the Data Source URL. The Request URL I’m using doesn’t return that information. Any ideas? 

242
j.botterill
Influencer
akeeri
Contributor
akeeriContributor
asked in Transformers
Convert dwg to shape by colors

Sometimes when I do a transformation from dwg to shape everything is included in 1 polyline (text, pipes, arrows) and when I look at it in inspektion mode I see that the dwg is splitting the lines by color, how do i do the transformation with a separation by color or adding the color value inside the whole polyline Sorry if my vocabulary is wrong

466
nielsgerrits
VIP
K
katrinaopperman
asked in Transformers
How to Use Summary Output from Statistics Calculator as part of calculation

Hello,This should be a very simple thing, but I just can't figure out how to do it!I want to append two tables together (Tables 1 & 2), both with very similar attributes, but I want to use the ID from Table 1 as the official Primary Key, and then get the highest number from that PK and increment upwards to create new PK values for the Table 2 appended values. I can get the maximum value from Table 1 using StatisticsCalculator, but I can't for the life of me work out how to use that value in the creation of the new Primary Key.I've been trying to do a conditional value calculation in Attribute Manager, something like, if Table 1 ID has a value, then PK = Table 1 ID, else use MaxValue from StatisticsCalculator and sum it with @sum(@double(@Count()),1.0) to increment up. But it's not working.Any ideas greatly appreciated!Thank you!Katrina

2395
jtorre5363
georgiec
Contributor
georgiecContributor
asked in Transformers
Difference between SPHERICAL_MERCATOR AND SPHERICAL_MERCATOR_0

I am Featurejoining the output from a change detector that compares an original layer from ArcGIS Online to a revised layer from my Geodatabase.    Note:  The AGOL layer was originally published from ArcGIS Pro from the coordinate system WGS 1984 Web Mercator (auxiliary sphere) .The Featurejoiner transformer rejects the features saying: “Coordinate system does not match first feature's coordinate system”Similar to the issue as seen in https://community.safe.com/topic/show?tid=20659&fid=8When I inspect the original and revised features, my features on AGOL have “SPHERICAL_MERCATOR” and my revised features have “SPHERICAL_MERCATOR_0”In  my workspace, my revised features are reprojected with ESRIReporojector to WGS_1984_Web_Mercator_Auxiliary_Sphere before entering the change detector. They then show a Coord system of  “SPHERICAL_MERCATOR_0”.    FYI  The ESRIProjector does display a warning saying “2.1_EsriReprojector: Ignoring input feature coordinate system `_MGA2020-54_FME_0'. This

341
georgiec
Contributor
nicholas
Contributor
nicholasContributor
asked in Transformers
How to read a JSON file from waze

I have a JSON file from waze that I am trying to read.I am using the JSON (JavaScript Object Notation) reader.There are three schema; alerts, irregularities and jamsI can read alerts okay with this queryjson["alerts"][*]Flatten Nested JSON values into Attributes = YesRead Entire Feature as JSON Fragment = Nowhich reads 433 alerts, each with a location.x and location.y which allows me to built a point object (Vertex) for each feature.But I am struggling with the irregularities and jams.The irregularities have a line object which obviously does not convert into two neat columns.How do I read the line object?I have attached the json file here

1493
desiree_at_safe
Safer

Badge Winners

  • FME Flow Certified Professional
    rchoucrounhas earned the badge FME Flow Certified Professional
  • FME Certified Professional
    aaron_chaputhas earned the badge FME Certified Professional
  • Integrate Data with the FME Platform Training
    flmuenznerhas earned the badge Integrate Data with the FME Platform Training
  • Teacher (Grey)
    bmoody81has earned the badge Teacher (Grey)
  • Webinar Watcher (Grey)
    gruelle88has earned the badge Webinar Watcher (Grey)
Show all badges

Community Stats

31,327
Posts
118,852
Replies
38,954
Members

Latest FME

Terms & ConditionsCookie settings
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.