Skip to main content
  • EN

Transformers

Using transformers to modify data.

  • 12,738 Posts
  • 50,954 Replies
12738 Posts
messagemauri
Contributor
messagemauriContributor
asked in Transformers
HTTPCaller transferred a partial file

Hello,I’ve had a workbench running on FME Server and FME Desktop 2022.2 perfectly fine through the last 2 years. Suddenly, it’s throwing an error HTTP Transfer error: Transferred a partial file. I tested it with FME Server and FME Desktop 2022.2 and get the same error. When I access the url through a browser, it perfectly downloads the zip file. But the HTTPCaller fails. Request Url: https://static.aer.ca/prd/data/wells/PLA_RTF_SHP.zipAny insights would be greatly appreciated.Thank you. Sample below for testing purposes, attempting to download the file in my local.HTTPCaller settings:  Error: 

41
ctredinnick
Supporter
5 minutes ago
lifalin2016
Contributor
lifalin2016Contributor
asked in Transformers
2024.2 yields an error in SharepointOnlineConnector ?

Hi.I built a workspace in 2024.1, that fetches a number of files from a customer Sharepoint site. It works (in 2024.1) without problems.I just installed 2024.2, and ran the workspace (with no changes).Now it issues an error with a SyntaxWarning from the package “safe.microsoft-sharepoint”:C:\Users\xxx\AppData\Roaming\Safe Software\FME\Packages\24783-win64\python\safe.microsoft-sharepoint\stringcase.py:247: SyntaxWarning: invalid escape sequence '\W' return re.sub("\W+", "", string)Is this something that I should worry about ?SharepointOnlineConnector is not listed as an upgradable transformer in 2024.2Cheers.

659
chriswilson
Supporter
50 minutes ago
sandanuwan
Contributor
sandanuwanContributor
asked in Transformers
Microsoft SQL Server Spatial Reader/Writer Supported FME version

Which version of FME Desktop/Flow supports “Microsoft SQL Server Spatial Reader/Writer”? I want to re-project MapInfo Tables in the GIS database. Currently I’m using FME Desktop 2022.1. Is there any alternative transformer I can use to re project the objects in the live database? 

192
sandanuwan
Contributor
1 hour ago
vxn43
Supporter
vxn43Supporter
asked in Transformers
path_modified_date and Directory and File Pathnames Reader

What is the best way to extract the path_modified_date, path_created_date, and path_accessed_date for files in a folder (these are shapefiles and zip file)? I ran the Directory and File Pathnames Reader, and it shows these attributes as missing. Is there a transformer I need to use after this Reader to extract this information?  

101
david_r
Evangelist
7 hours ago
juneacosta
Contributor
juneacostaContributor
asked in Transformers
Help Configuring FME Geocoder with Custom Locator on ArcGIS Portal

Hi everyone,I’m trying to set up FME to geocode addresses using a custom locator hosted in ArcGIS Portal, but I’ve hit a roadblock.In the Geocoder transformer, one of the parameters is the Authentication Type, where I selected Web Service (see attached image). From there, I’m required to create a web connection. This is where I’m having trouble. I’ve been following this guide, which explains how to connect FME to an ArcGIS Portal locator. The guide seems to focus on locators tied to an "Application" type item, but our locator is a hosted locator, not an application.  Here’s where I’m stuck:  I’m using the **Geocoder** transformer in FME and trying to configure it to connect to our hosted locator.   The guide asks for information like a client ID and client secret, which seem to apply to an Application type item in Portal. Since our locator isn’t an application, I’m not sure what to provide for those fields.  Can someone confirm:  Is it even possible to use a hosted locator from ArcGIS

70
juneacosta
Contributor
18 hours ago
miguelhacar
Contributor
miguelhacarContributor
asked in Transformers
In the exercise: "Add Appearances to a 3D City Model" I have a problem with the "AppearanceSetter"

In the exercise: "Add Appearances to a 3D City Model" I have a problem with the "AppearanceSetter" transformer because I cannot get the colors in the buildings in the PDF output, as described by Liz Sanderson. I always get them in RED in both cases, namely: "AppearancesRed" and "AppearancesBlue". I have run the workspace as I downloaded it but both types of buildings always come out in RED in the PDF.https://support.safe.com/hc/en-us/articles/25407543263373-Add-Appearances-to-a-3D-City-Model

233
kailinatsafe
Safer
1 day ago
moustik
Contributor
moustikContributor
asked in Transformers
SQLExecutor in FME2024.1 doesn't show Geometry of PostGIS raster, in contrary with FME2023.1

Hello,I am currently working with some database PostGIS rasters and wanted to import some into FME 2024.1, using a filter to only select 2 out of 11030.To do so, I tried both a SQL Executor and a FeatureReader (as both enables PostGIS raster reading).The SQL Executor filters the rasters correctly but doesn’t create the geometry and returns an error in the log file.The FeatureReader loads the 11030 rasters without filtering, suggesting a problem with the “WHERE clause”.I tried the same thing using FME 2023.1 and the SQL Executor is working under this version. I am wondering if there is a bug in the FME 2024.1 version.The problem was the same for the Feature Reader.A colleague said that the Feature Reader outputs everything as well in previous versions of FME (2022.2 and 2021.1).Do you know when the FME 2024.1 bug will be corrected?We are linking our work with ArcPro and we know we will have to use FME 2024.I am enclosing a share folder with the different logs (as txt) and the 2 workbenc

804
moustik
Contributor
1 day ago
olyster
Contributor
olysterContributor
asked in Transformers
Prevent Logging of authorization header with HTTPCaller

Hi,When using HTTPCaller with an authorization header with a bearer token, the token appears in the log.How can I prevent this ?I’m using FME desktop and server version 2021 so LOG_FILTER_MASK doesn’t work. Updating the version is not a solution. The bearer token is obtained at runtime so I can’t put it in a private parameter (PP cannot be updated at runtime).Thanks.

449
olyster
Contributor
1 day ago
mzebrows
Contributor
mzebrowsContributor
asked in Transformers
How to fill CellName attribute while exporting data to dgn

Hi,I have two Oracle tables: POINTS_TABLE and LINES_TABLE. I’m trying to export both tables to DGN.  I’m using “Bentley MicroStation Desing (V8)” writer. In writer parameters, I filled out “Cell library file”, and now I’m trying to do something to fill “CellName”. How should I do it, to have different “CellName” value for POINTS_TABLE and for LINES_TABLE in my output dgn file?

70
mzebrows
Contributor
1 day ago
lifalin2016
Contributor
lifalin2016Contributor
asked in Transformers
XML flattening of lists with one entry

Hi all.I'm having a hard time figuring out why lists with one entry, especially if it's read from XML, is mishandled in FME.A list may have any number of entries, and if reading such from XML, they often have any number including zero entries.When flattening the output, cycling thru the list, one normally uses ListExploder. However, if the list in question has one entry, it's auto-exploded, which makes ListExploder reject it. So I have to check the rejected features to see, if the list had been auto-exploded or if it had no entries at all. E.g. two or more transformers instead of just one.Why?Wouldn't it be more logical to _always_ create a list, if there are 1 or more entries ?It works as such in 2021.0Is this behaviour specific to XML (or JSON?) flattening ?Cheers.

15413
nielsgerrits
VIP
1 day ago
ahuarte47
Participant
ahuarte47Participant
asked in Transformers
Apache Parquer Writer does not write DateTime columns properly

Hi community, I am using FME Workbench 2024.1 to create parquet files. Those files contain DateTime columns. It seems as those DateTime columns are missing a codec flag in order to enable the loading in other software (Dremio in my user case). Using parquet-tools we can see the problem:java -jar parquet-tools-1.9.0.jar schema ~/Downloads/part0.parquet can not read class org.apache.parquet.format.FileMetaData: Required field 'codec' was not present! Struct: ColumnMetaData(type:INT64, encodings:[RLE_DICTIONARY, PLAIN, RLE], path_in_schema:[_transaction_ts], codec:null, num_values:66, total_uncompressed_size:95, total_compressed_size:113, data_page_offset:35, dictionary_page_offset:4, statistics:Statistics(max:EC 90 D1 36 42 42 10 18, min:EC 90 D1 36 42 42 10 18, null_count:0), encoding_stats:[PageEncodingStats(page_type:DICTIONARY_PAGE, encoding:PLAIN, count:1), PageEncodingStats(page_type:DATA_PAGE, encoding:RLE_DICTIONARY, count:1)]) I include here a small parquet example with two date

263
virtualcitymatt
Celebrity
1 day ago
danraleigh
Contributor
danraleighContributor
asked in Transformers
Calculating Cumulative Distances for Different Groups of Points

I have a workflow that assesses the proximity of points along lines. These points extend along a series of line segments of varying lengths. For instance, the distances to the next points might be 0ft (the start), 100ft, 100ft, 200ft, and 250ft, so I would like for the corresponding field next to these values to display 0ft, 100ft, 200ft, 400ft, and 650ft, etc. I am aware of the workflow to find cumulative sums (Attribute Creator>Statistics Calculator>Attribute Renamer). While this processes as described, I regret that the cumulative sums do not “reset” when there’s more than one batch of point data within the feature table (the initial point of an additional batch of points simply takes the prior set’s end sum and begins adding new distances from there). If there’s a way to prompt the cumulative sums to account for new “initial” points in a feature dataset and so start from scratch each time, I would greatly appreciate any insight on how to prompt that to occur. I’m happy to sha

242
danraleigh
Contributor
2 days ago
danraleigh
Contributor
danraleighContributor
asked in Transformers
Creating Points Along a Polyline at Equal Intervals

I am hoping to produce a group of points along polylines of various lengths. I’ve been trying to mimic the GeneratePointsAlongLines() method from arcpy and ArcGIS Pro which will simply build out points from the start of a polyline at a designated interval (regardless of line segment vertices). I’ve investigated Densifier, but this transformer incorporates vertices and produces points that do not conform to the designated interval. I’ve contemplated utilizing Snipper, but this seems challenging given that many of the line segments I expect to incorporate may vary in lengths for different inputs.I’d appreciate any insight that can be offered.

292
ebygomm
Influencer
2 days ago
saravanan
Contributor
saravananContributor
asked in Transformers
Modify Schema list attributes

I have a requirement to modify the list attribute generated on schema port.Lets say my schema scanner have default attributes name, fme data type and native data type.I want to look on name attribute and find any of its elements have OBJECTID string and remove if it is present.I try to modify list attributes using python caller. It modifies the list elements when I overwrite using feature.setAttribute method it is not working and returns original schema attributes unmodified. Any help using python would be highly appreciable. 

334
saravanan
Contributor
2 days ago
katt
Contributor
kattContributor
asked in Transformers
Finds arcs in a polygon

Hi, I have a lot of polygons. I want to identify these ones who contain arcs.option: the polygon is made of lines onlythe polygon is made of lines and arcsI want to filter out all the polygons that contain lines only (option 1). I want to keep the ones that contain arcs (option 2). I tried the geometry property extractor but all polygons are always rejected : MISSING_GEOMETRY_PROPERTY as the error message. I don’t understand why. pls advise. 

291
david_r
Evangelist
2 days ago
maphew
maphew
asked in Transformers
"Failed to free sufficient memory" log message

"Failed to free sufficient memory to reach the process usage limit"This warning is appearing my log files. What do I need to do about it? What is the potential impact if nothing is done?As far as I can tell there is nothing wrong with the output files the script is creating.  

54017
mattwilkie
Supporter
3 days ago
mgrimnes
Contributor
mgrimnesContributor
asked in Transformers
FeatureWriter and Esri ArcGIS Server Feature Service with dynamic schema fails when layername contains space

I have logged a support case about my problem, but figured I might as well check to see if anyone else have encoutered this problem:I am writing a workspace to geosynchronize a Feature Service with a database. The Feature Service contains about 30 layers that need to be updated, so I have put my bet in for a dynamic workspace, since the operation is the same for every layer. This saves me a lot of transformers and makes the maintenance of the workspace much simpler.My challenge here is that both the FeatureReader and FeatureWriter requires the name of the layer to be passed in as a parameter, not the layerid, which would not be a problem. Almost all of these layers have names containing, at least, one space, and this is what is causing my trouble.On the reader side, I can enclose the layer name in double quotes, like “My Layer”, and this works without problems.On the writer side, however, things are messed up. I have so far tried three different approaches:Enter the name “as is”, like

1368
mgrimnes
Contributor
3 days ago
so_much_more
Supporter
so_much_moreSupporter
asked in Transformers
Read .log files and check for errors

I have a work process that inlcudes running FME jobs daily.I would like to automate logging of potential errors. To do so i am thinking of just reading the .log files from the workbench(es) and search for error as keyword to catch if something went wrong in the translation. I thought it would be straight forward to implement, but i cannot find a way to read .log files using FME directly. Instead I am thinking to use a python script. But I wanted to ask here if I have missed an easy method to check log files for errors.

171
virtualcitymatt
Celebrity
6 days ago
canderton
Participant
candertonParticipant
asked in Transformers
How to Iterate Multiple Feature Classes and Shapefiles Through Transformer

Hi,I am using FME Workbench 2024. I am wanting to know if I can use a loop or iterative process of some sort to individually run multiple feature classes and shapefiles through a single AttributeKeeper Transformer for each data source, possibly using the names as an iterator. I am working on bringing in utility data from multiple sources and ultimately writing them to a single geodatabase feature dataset. First, I want to clean up the fields and only keep certain ones. Each source of data has a different schema. Currently, I can only use an AttributeKeeper on each feature class/shapefile in order to later output each to its own feature class in the output geodatabase. This would make for a messy workspace with many of the same transformers. Any advice would be greatly appreciated!

512
virtualcitymatt
Celebrity
6 days ago
syedh2993
Contributor
syedh2993Contributor
asked in Transformers
Create sub-element to every element of the list

Hi All,I am working with lists and require some guidance on the below scenarioI have list which contains 'n' elements. I would like to add a sub-element to each element in the list.Assuming my list name called as "list{}"Example: (My list currently looks like this)    List{0}        element1        element2        element3    List{1}        element1        element2        element3    List{2}        element1        element2        element3    List{n}        element1        element2        element3        I would like to add new sub-element called "element4" at the end of each list element (i.e., add "element4" as sub-element below "element3" )        Expected Output (I would like to achieve something like below)    List{0}        element1        element2        element3        element4    List{1}        element1        element2        element3        element4    List{2}        element1        element2        element3        element4    List{n}        element1        element2        elem

221
geomancer
Evangelist
8 days ago
kenortey8
Contributor
kenortey8Contributor
asked in Transformers
GML LOD2 To point cloud

I’m at working on matching indoor point cloud data to gml LOD2 data using ICP so I want to transform my gml data to a point cloud data to help ease the matching. Can anyone please help me? question is how do I transform GML LOD2 to point cloud data? also if there are other suggestions to complete this task. Thank you 

414
danilo_fme
Evangelist
14 days ago
mahapandian
Contributor
mahapandianContributor
asked in Transformers
Folder and data segregation and validation in FME form

Hi,I have some villages (Primary and secondary villages), each containing a main folder, a subfolder, a data folder, and some empty folders. I will designate one village as the primary village. All sub-village files/data will be stored in the main village folder. Only location path links will be kept in the secondary village folder (marked in red color). Other data will be sourced from different locations (Marked in green color). Finally, I must validate whether all folders and data have been correctly segregated or not. I am currently performing this process manually, which involves copying, pasting, and validating each folder and village individually. This manual process is quite time-consuming and manpower. Kindly help me about this test cases and workflow. I am new to FME and am eager to learn more about its functionalities. Thanks.

652
mahapandian
Contributor
16 days ago
doug_herrick
Contributor
doug_herrickContributor
asked in Transformers
The object at location ___ was previously deleted and cannot be retrieved from the Object Stash

When running an old script in FME 2024.1.1.0 we get 58 errors “The object at location ___ was previously deleted and cannot be retrieved from the Object Stash” (see attached log file).  It doesn’t appear to affect the output, the log file says “Translation was SUCCESSFUL” and CPU and Memory usage don’t get close to 100%.  Can we ignore these errors?This is part of a much larger script but only this part is producing the errors. We updated the transformers and saved the script to FME 2024 but still get the errors at the transformer FeatureJoiner_13.  The only way to not get the errors is to run the script up to the 2 preceding transformers (“Run to this”) and then run FeatureJoiner_13.  The output appears to be the same as when we were getting the errors.  We tried adding in FeatureHolders before FeatureJoiner_13 but still got the errors. ByeDoug

954
doug_herrick
Contributor
18 days ago
davncini
Contributor
davnciniContributor
asked in Transformers
Distribute point within polygon

I have a point feature which has been developed based on the polygon ID. All of these points are created at the polygon Centroid, I want to distribute the point randomly within the polygon

645
david_r
Evangelist
20 days ago
malthern
Participant
malthernParticipant
asked in Transformers
How do I add a folder with FME Packages SDK

I have been using the FME Packager in Python to create my own transformers. I am trying to add an “images” folder to my package that my transformer will use however I get errors whenever I try to package my project.This is my package.yml file, Ive been using the Hello World Package guide for references and trying to look up ways to do it This is my file structure This is the error I get  

311
debbiatsafe
Safer
20 days ago

Badge winners

Show all badges
30,414
Posts
Terms & ConditionsCookie settings
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.