Transformers

This is more of a hypothetical question, I don't have a problem that needs solving, but am pondering the possibilities and limits of FME.What are the strategies to process huge datasets? I would imagine that there is a limit to the resource management that FME can do on its own? At some point the software might need some "help" in order to break down the data in manageable chunks? How is this done? Tiling? Memory management? Renting a super computer? Etc?Trying to think of an example I came up with the following idea. How would one go about generalizing the contour lines for a whole country in one run without watching life go by? Please note that I am not asking about how to generalize, my question is what strategies there are to speed up huge processing operations similar to the example above?

I have a workspace where I have lines that are clearly duplicated but the matcher does not identify them because the coordinates are flipped on one versus the other. Is there a way to identify duplicate geometries for lines regardless of whether the coordinates are flipped. For example: Line 1Coordinate 0 = 1973175.975504196, 1785187.3988771015Coordinate 1 = 1973164.445329631, 1784950.337012621 Line 2Coordinate 0 = 1973164.445329631, 1784950.337012621Coordinate 1 = 1973175.975504196, 1785187.3988771015 Both lines are identical, however the direction of one is different from the other. So how can I eliminate one of these based solely on geometry.

A tool in FME to remove duplicates,  I couldn’t figure out which tool can remove duplicates. For example, in case we have two lines that are identical (one on top of another), then which tools can remove this duplicate?  For example, in the screenshot below we have 4 lines that are identical, then how to remove three of them and to leave one?    Thank you  Best  Jamal

Dear FME community,I have a question concerning the FME workspace runner. I have a workspace with 5 different inputs. The goal is to create geospatial PDFs. Each of the inputs represents a different layer of the geospatial PDF. Now I know that the workspace runner will import the inputs once at a time. The problem is now that in the 5 input-folders have each a different number of files. This means that at the end not every PDF will have the same number of layers. How can I guarantee that the workspace runner selects the right inputs? I have named the input files exactly the same, but they are in different folders. So it would probably be easy to just compare the base names.  I would be very happy, if somebody could help me with this. Thank you so much in advance! With kind regardsChristian

HelloI am having trouble extracting attributes from a kml for the features that have photos.I am using XQuery similar to this example https://community.safe.com/s/question/0D54Q0000848CTkSAM/extracting-information-from-different-folders-in-kml-to-convert-to-the-shapefile That method works for the features that don't have photos but the features that have photos seem to have a extra section in the HTML that is causing my XQuery to fail. The Xquery I am using is : declare default element namespace "http://www.w3.org/1999/xhtml";for $x in /html/body/table/tr/td/table/trreturn fme:set-attribute($x/td[1]/text(),$x/td[2]/text()) I am pretty sure I just need to modify the Xquery 

Hi. Create a pyramid. Easy.How I could extract only the most dense level without processing all others levels?

The features are a subset of US Census tracts, and a handful of them have multi-part geometries that do not share a boundary. My goal is to treat these multi-part geometries as a single unit based on the unique ID (FIPS code) for each tract.

Hi,The problem I'm trying to solve is as follows:I have about 2,000 aerial imagery ECW files, averaging in size around 30MB (so around 60GB in total).Each of the files cover 1km x 1km on the ground at a resolution of 10cm, so are 10,000px x 10,000px.I have been asked to downsample each file to 2,048px x 2,048px and output the results as a series of 4km x 4km tiles (each tile being 8,192px x 8,192px).I have an index feature class that lists each 'FileName' and its corresponding 'Tile' attribute.I've attached an image that shows this spatially, below.Large black squares and number labels represent the tiles that I want to output. The coloured squares with red outlines represent the original files.Previously, when working with a much smaller amount of data, I've been able to use my index feature class, in conjunction with a StringConcatenater, to provide the full path of each file to an 'ER Mapper ECW' FeatureReader.From there, I used RasterResampler an

Have two workspaces that when I run it under my account or the service account on the server, it runs fine, published to FME Server and set up automation. It APPEARS to run (one workspace checks for the latest zip file in a directory on a file share and then copies to the resources directory and unzips on the server - this works as it should under both accounts and in FME server when run separately thru Run Workspace). The second workspace is to process the data that was unzipped and place in the resources directory - the files are separated into two directories due to how the vendor supplies the data for download - the script will read this data via either account. It will operate as expected when run with Workbench. Running via Server it processes, but when I look at the output data, not all of it is there! The csv files appear to be appending to the tables in the SDE. The graphic data doesn't even show that it attempted to write to the feature class. I'm stumped. The graph

Hi everyone, I am trying to download a set of files with the AutodeskBIM360DocsConnector and save them to a temporary location. The temp location is generated with the TempPathCreator. The BIM 360 connector does not have the option to pick an attribute for the "Download to Folder" parameter. Typing out @Value(DownloadLocation) seems to trigger FME to recognize that I want to use an attribute value, however during translation the attribute value is ignored and instead FME uses the string I typed in i.e. it creates a map named "@Value(DownloadFolder)".   

I have text where spaces between some words is missing. These spots are easy to find with regex because it is always something following by capital letter without the space. I can't workout how to add the space though. My String Replacer looks like this: @ReplaceRegEx(@Value(Speciality),([^ ])([A-Z]), \\1, \\2) The output it produces is:Anxiety, Panic & Phobia shildren & Adolescent selationships Outside work (ndividual counselling )eadership Original:Anxiety, Panic & PhobiasChildren & AdolescentsRelationships Outside work (Individual counselling)Leadership So it adds space, just not where I need it and removes the capital letter. I tried many variations, but none of them produces the needed outcome. 

My flow includes a looping Custom Transformer that gets records from a paginated API until there are no records left to get - unfortunately the API doesn't have a "last page" flag or anything, so the only way to check that the flow should end is to keep trying until we get a 400 Bad Request status code with a message like "Request out of range". When this happens, the loop should break and output all the features that have been created up to that point. At the moment, the only way I can get this to break is with a Terminator, which is (a) making the flow look like it failed, and (b) not allowing me to continue with the next steps of generating a HTML report. Is there a more graceful way to exit a loop? The transformer looks like this:The relevant part in the flow itself looks like this. The Tester is doing exactly what it's supposed to, but it doesn't seem to be reaching the HTMLReportGenerator.

Hi FME Community,I have converted my Excel file into a CSV file. I would like the heading to be automatically deleted.In the picture I have shown with red marking which part has to be deletedI would be grateful if you could tell me whether there is a solution.Thanks in advance

Hi,I have a scripted python parameter which generates a unique id. This parameter is used in a lot of SQLexecuters to to create and insert into temporary with this id in the name of the table and such. When I run the workspace it works fine, but when I put feature caching on the value of the parameter seems to change each time it's used in a SQLExecuter. Is this a knwn issue? Is there a simple work around?I use FME 2021.1.3.0. I don't know if it worked before. Thank you,Arno

Hello, so I want to find a missing value as all the "Unit" must be in sequence in each floor.​Here are my rules:All the units should be assigned sequentially to the corresponding floor, meaning 1/F: Unit = 0001, 0002, 0003, till to the largest number in the data2/F: Unit = 0001, 0002, 0003, till to the largest number in the data​I have "FloorNo_FromFloorPolyID", meaning the level of floors."Test_Unit_No" meaning the numbering of the unit.​I want to find out the missing unit number of each floor as the input data will have a lot of floors in the future. As the sample shown below, Floor 10 is missing Unit 0001 to 0004.​Could someone please give me a hand. Should I use python caller? Yet I have no clue how to write script ... please help.​Thank you in advanced.

How convert to complex XML to creating relational database tables ? using xml hieararchy

I have xml file and convert relatinal database tables for using xml hierarchy(parent-child) so is there any automation tool for this process ?

Hi all!My CAD layer is a mapimport from shapefile. So, it basically still has attributes when I converted to a layer with point type. When I wanted to export again to shapefile through FME, I don't know how to retain those attributes.  In CAD, actually you can use MAPEXPORT to retain attributes, but then it will be time-consuming if you need to export one by one when you have many.  So guys, do you have a solution for me to solve this problem? Thanks in advance!

Hi all,I have a CSV which I read into FME desktop. It is bore hole data with attributes:site: Adepth: 0depth: 20depth: 30end depth: 35 I want to select the max(depth) for each 'site' and update that value to be 'end depth', then proceed with other transformations. Note sure what transformer to use to do this?I've looked at Tester, inlinequerier...help ? thanks

Hello everyone,​So I want to do something similar to "IMDF Validator".​My data format is "GeoJSON" in one single file. When I import the data, it will break into 4 feature types. ​I just not sure how to set up something similar to IMDF validator, with "Creator" at the front (just like the picture I uploaded? Also, the user parameter of IMDF validator is set default in ZIP file, yet I have to set it to geojson file.​Should I use Create Custom Transformer? But I am not sure how to set the parameter. It is because my data will be getting more and more complicated. I have to make it more tidy first.​Thank you in advanced. Zach

I'm trying to import a text file without headers to FILEGDB while using the SchemaMapper Transformer to rename the columns. I've followed tutorials and articles but am not successful. Columns aren't being renamed and when I run the workspace no features are written.Any tips would be greatly appreciated. Here are screenshots of my parameters and workspace: Schema Map (CSV):SourceAttribute,DestinationAttributecol0,TYPEcol1,STRAPcol2,SECTIONcol3,TOWNSHIPcol4,RANGEcol5,MAP_IDcol6,CONF_FLGcol7,ADDR_1col8,ADDR_2col9,CITYcol10,STATEcol11,ZIPcol12,DOR_CDcol13,SUBcol14,DSCRcol15,PLAT_BKcol16,PLAT_PGcol17,NH_CDcol18,PCT_SOHcol19,INC_FLGcol20,BASE_YRcol21,ASD_VALcol22,SOH_VALcol23,BLD_VALcol24,JST_VALcol25,TAX_DISTcol26,TAX_DISTcol27,TAX DISTcol28,MILL_CDcol29,SQFTcol30,ACREAGE

So I have multiple columns of data that return a yes or no value. I'm really only concerned with the yes value.  When the value in the Cut Grass attribute is 'Yes' - i want to replace the Yes with say "cut grass to 100mm"Then i have another Yes value in Remove Flammable materials column and I want to replace the Yes with "remove materials".  Is there a multiple string replacer transformer that I can use.  I need to do this for approx 13 different attributes. I have read about string replacer or attribute value mapper. But whilst string replacer can replace multiple attributes to the same replacement text it doesn't allow for differing replacement text individual to a column.

I have set up a python script to run a workspace through the FME Python API using FMEWorkspaceRunner. Python is throwing an error called 'FMEException: 1: Failure running workspace "<my path.fmw>" '. I did some troubleshooting by removing the HYPER writer and replacing with a csv writer, then a gdb writer. Both of those writers worked fine. Also, the workspace works fine without going through the API. Something about the HYPER and the API. Any suggestions or known issues? Thank you, TylerFME 2021Python 3.7

I have time-series data collected at 5 minutes intervals and I would like to summarize or average it to the daily interval. How do I go about it in FME? Thanks