- - Community Resources
Ideas
Product Updates
Events
Blog

Home
Forums
FME Form
Data

Data

Using readers and writers to extract and load data.

6,781 Posts
24,934 Replies

When you subscribe we will email you when there is a new topic in this category

6781 Posts

Recently active

Most replies Most views Newest first

ravibansalContributor

asked in Data

How do I write Lat Long to a non-spatial table with XY coordinate?

I have a non-spatial table with about 5 million records of XY coordinates. I need to update the table with the Lat Long values in the so-named columns.I am facing problems with writing these to the table. When I choose UPDATE/UPSERT/fme_db_operation, the writing takes a seconf for every 10 records. It’s impossibly slow.When I choose, Insert with DROP and CREATE or TRUNCATE EXISTING, the script gets stuck with following last 4 lines in the log: Writer `MSSQL_ADO_2' of type `MSSQL_ADO' using group definition keyword `MSSQL_ADO_2_DEF'Microsoft SQL Server Non-Spatial Writer: Opening `GISReporting' for write operationMicrosoft SQL Server Non-Spatial Writer: Read 1 DEF line(s). Found 9 attribute(s)Microsoft SQL Server Non-Spatial Writer: Trying to connect using Microsoft OLE DB Driver 19 for SQL Server...MSSQL_ADO writer: Splitting bulk features into individual features NOTE: I have set BULK Read and Write at 100000 records for every transaction.

373

7 months ago

albjohContributor

asked in Data

Dynamic truncate writing to SDE

I’m having trouble writing dynamically to a SDE feature dataset using the ‘truncate’ option in FME 2023.1. The features are written, but there is no ‘truncating’ taking place before the writing and instead the data doubles up... It works as expected in FME 2021.2.6 however, maybe due to the different way in which the feature dataset is specified in this version under ‘Table Creation Parameters’. How do I make it truncate also in 2023.1 version? Current set up in writer for FME 2023.1:

1427

7 months ago

franco69Contributor

asked in Data

how to extract a certain range of number from an attribute?

Hi all,i know it will maybe an easy one….I’ve tried with substringextractor or with AttributeTrimmer as well but i couldnt reach the right result.i have attribute like these for example:he0_wasser_WSG_TK_20143_AGE_f_r01_231030 orni0_geobasisdaten_gemischte_industrie_gewerbe_flaechen_20256_AGE_f_r01_240118i have to split in this case “he0” and “ni0” and write that in an attribute xi have to split “wasser” and “geobasisdaten” and write that in an attribute yi have to split “AGE” in an attribute…….and so onnow comes the challenge:the text between “wasser” or “geobasidaten” and the 5digit number (20143, 20256) has sometimes 2 words, sometimes 3 or 5 words…..but i have to write the whole text in an attribute z….ho can i do that? i am not so well in regexp…..so i ask here for help Thank youCheersFranco

907

7 months ago

charryContributor

asked in Data

FME cant read ESRI gdb,but can writer

515

7 months ago

arnaud.leidgensContributor

asked in Data

ArcGIS pgdata storage issue

Hello everyone,For one year, I’ve been running different workbenches daily (Windows Scheduler), reading from MSSQL tables to write into various hosted feature layers, used in various dashboards. I noticed an unusual consumption of the storage on our GIS server, coming from the “pgdata” folder of Arcgis Entreprise, so the underlying postgresql database (up to 20go additional Gb per month, while the size of all my hosted features stay constant at around 1gb only in total). After several tehnical sessions with ESRI, there is no doubts that this is caused by my FME workflows. Those workbenches are far to be optimised (as I was only starting to use FME) but they are working fine. It was easy to truncate all features to insert new features, sometimes up to 150 000 features daily, for one layer (see below, simplified version). The inserted features are polygons, from the same source hosted feature layer, daily joined with most recent MSSQL data. NB: Objectid is removed with “attribute manager

251

7 months ago

solyContributor

asked in Data

how to find all point and cables in different paths between start and end point ?

Hello , i have too many points I would like to find the points and lines between start of path and end of path in different paths ?i would like to have excel file that include First row Start point (a) - next point (b)- lines (line id 1)Second row From b point - next point C - lines (line id 2 )and so on till end point (blue point )fllow direction from start point to end point as picture fme 2021 thanks in advance

1627

7 months ago

fmemapperContributor

asked in Data

ArcGIS Online Feature Service Writer not Adding or Updating Records

I have followed this tutorial:Updating ArcGIS Online Feature Services using UPSERT (safe.com)As the last screenshot shows, I have 1495 records. Comparing the original CSV used to do the initial data load and the updated CSV, the new records in the updated CSV have not been added and existing records with changed attributes have not been updated.The tutorial suggests this is possible using the UPSERT operation. Example update I expected with the highlighted text the changed attribute value: 1000,ZYX3116631180,-123.093432,49.281221,31180,-53.7761,-21.7304,1,2011-12-02 22:01 Example add I expected: 9000,ABC1069610710,-123.093414,49.281637,10710,-79.377,-17.3218,1,2021-12-06 1:41

1013

7 months ago

pgammieContributor

asked in Data

Creating a File Geodatabase with Feature Datasets FME 2018

I am creating a workbench where I will bring spatial object assets from an oracle database into a file geodatabase. I want to sort the assets into feature datasets as I go but I am confused as to how to go about this. I have only done a few FME workbenches so I don’t have a great deal of experience. So far, when I have been creating File Geodatabases, I add a Esri Geodatabase (Open API) writer and then the file geodatabase is created and any features I connect to that writer are added in. So its kind of a two for one where the file geodatabase is created plus my features are converted and added to it. How would I go about creating a feature dataset during this process? Is it possible with FME 2018 or does that come with newer versions. We will not be upgrading for several months and my co-op term will be over by that time. Thank you!

581

7 months ago

jeroen_hackContributor

asked in Data

Concatenate values in groups of 5

Hi fellow FME enthousiasts,Another case i'm struggling with today.I have a list of parcels that need to be added into a Excel template. In most cases I can sort the parcels by parcelnumber and concatenate them grouped by the Section number and write them to the Excel.But the field in the template can fit a maximum of 5 parcel numbers. So if there are more then 5 parcels they need to be split in groups of 5. So say i have a file with 13 parcels, I would need to split them into 3 groups (2 of 5 parcels and 1 of 3 parcels). I can get the number of parcels with a StatisticsCalculator, but i struggle with the part where i have to split them into groups to write to the Excel. How would you do this? Cheers!

362

7 months ago

nielsperplaceContributor

asked in Data

Complex issue: Need to merge features of 1 layer based on overlap with 2nd layer

The steps I have so far: 2 layersSpatialRelator, which generates the list attributes with the intersects.Then I test the output to find out which features have more than 1 relation (indicating overlapping with more than 1 feature).Then I explode that list and delete the unimportant attributes with AttributeManager. I now have a table like this. First column containing a unique identifier for Layer 1, the second row containing the unique identifier for Layer 2. A feature of Layer 1 can overlap with multiple features of Layer 2, and vica verca, hence the duplicates on both sides.Layer 1 Layer 2 ID1 123 ID1 345 ID2 987 ID3 456 ID3 123 Now the hard part. I can FeatureJoiner them together, and then dissolve, but then I miss a lot of features and don't dissolve enough overlaps. It doesnt create the perfect output.What I want to achieve: I want all features of Layer 2 to dissolve together, based on the overlaps with features from Layer 2.I added a drawing to visualise wha

964

7 months ago

jeroen_hackContributor

asked in Data

Writing all records to two different formats depending on a Test

Hello fellow FME enthousiasts,I have a case which i can't seem to find a solution for.We have a process that adds a score to incoming records based on certifications added as hyperlinks in a table.During several steps these certifications have to be manually checked by an employee. Finally when everything has been approved the records are written to a central database (this can only be done when all the records have been approved, so the whole dataset has to be kept together during the whole process). Right now i have the process setup to output all data to an Excel table (with a template to make it readable for the employee). Here the employee can select if a certification is approved or not. This Excel table is then put back into the next process untill all necessary checks have been done.Each time a proces is run i would like to check at the end if all records are approved or not. Depending on the outcome two things can happenIf ANY are not approved, ALL records needs to be output t

253

7 months ago

galigisEnthusiast

asked in Data

Feature Join GDB with Attachments (JPGs)

Hi All,I need to run a join between an excel file to a feature class (GDB) with attachments by an unique ID. The GDB has been used to collect data within a Survey123 form.I did some testing but can’t retain the attachments in the final output,Any idea how to retain the attachments after running the join?

391

7 months ago

pgammieContributor

asked in Data

ESRI Reprojector - GCS WGS84 and PCS WGS84 Spherical Mercator

I am building a workbench that uses spatial filters, I have several readers input to the spatial filters that have different source coordinate systems. Originally, I used the ESRI reprojector to convert everything to 3TM (EPSG: 3776) prior to the spatial filters. This included a file with a source coordinate system of WGS84 Spherical Mercator (EPSG: 3857). However, my end result is about 4 feet off of where it should be. I tried selecting a transformation (I used Pro to see which transformation they used when I used the project tool and selected that one) but the result was still the same. I performed the projection in Pro on the same file and it was successful so it was a bit confusing as to what I was missing in FME. I decided to instead project my other files to WGS84 Spherical Mercator and work within that coordinates system. It was successful. I am guessing I may be using the ESRI reprojector incorrectly, or missing an input.

250

7 months ago

jdhContributor

asked in Data

Geotiff band offset and scaling

I’m reading a geotiff file with embedded scaling and offset, snippet from gdalinfo:Band 1 Block=1024x1024 Type=Byte, ColorInterp=Red NoData Value=255 Overviews: 6000x6000, 3000x3000, 1500x1500, 750x750 Offset: -1, Scale:0.008Band 2 Block=1024x1024 Type=Byte, ColorInterp=Green NoData Value=255 Overviews: 6000x6000, 3000x3000, 1500x1500, 750x750 Offset: -1, Scale:0.008Band 3 Block=1024x1024 Type=Byte, ColorInterp=Blue NoData Value=255 Overviews: 6000x6000, 3000x3000, 1500x1500, 750x750 Offset: -1, Scale:0.008When working with the data in FME just the the 8-bit value is shown (0-255) when I was expecting values between -1 and 1. In my particular use case the scaling is constant and I can just apply a correction, but I was wondering if there is a setting I’m missing to have FME apply the scaling/offset automatically.

490

7 months ago

becContributor

asked in Data

Trying to update fields in an AGOL layer and FME script keeps failing after first 3000 features

I am trying to update approximately 10 attribute fields in a point layer in AGOL using FME. When I run the script, it keeps failing and I’m not sure where the hang-up is based on the warning and error messages I am getting. Below is what I keep getting once my script has failed, with the ones in red being the error messages (the last two bullet points): Feature Caching is ON The workspace may run slower because features are being output and recorded on all output ports regardless of whether or not there is a connection. ArcGIS Online Feature Service Reader: Reading from from feature service 'https://services1.arcgis.com/9yy6msODkIBzkUXU/arcgis/rest/services/HII_2020_2022/FeatureServer' using filters defined in 'HII_2020_2022' ArcGIS Online Feature Service Writer: Feature Service 'HII_2020_2022' does not support geometry updates. If features contain geometry, the geometry will be ignored ArcGIS Online Feature Service Writer: Attribute 'OBJECTID' on feature type 'HII_2020_2022' is mar

32512

7 months ago

auspolmateParticipant

asked in Data

Convert a Geodatase to KML

Hi,I am hoping someone can convert a Geodatabase provided by the United Nations (Unitar & Unosat).It is published under creative commons 4.0.The map shows each building damaged in Gaza.I do not have any product that opens Geodatabsae files.I would be appreciative if someone could convert the file into KML so I can open the file in Google Earth.The zipped geodatabase file is 2.35Mb.The link is the location of the Geodatase file. https://data.humdata.org/dataset/unosat-gaza-strip-comprehensive-building-housing-unit-damage-assessment-march-2024#I have also attached two images.The first image shows the page with the GeodatabaseThe second image shows how the UN have used the file to present to the public.I have submitted a request to UNTAR however they have not responded.ThanksHarry Sullivan

491

7 months ago

vishnunv2696Contributor

asked in Data

Finding Start and End Names of Intersected Road Network Lines

Hello FME Experts,I'm currently working with road network spatial data, which contains a multitude of line features representing roads. Each road feature intersects with others, and I need to determine the correct start and end names for each road segment (i.e., the names of the roads at the start and end nodes).I'm wondering if there's an efficient way to accomplish this using FME. Specifically, I'm looking for guidance on:How to identify the start and end nodes of each road segment. How to extract the names associated with these nodes.Any suggestions, workflows, or examples would be immensely helpful. Thank you for your insights!Best regards,Vishnu N V

Cannot write to ESRI after using Aggregator (mutipart geometry)

Hi,I am creating some admin areas which feature multiple islands within each admin areas.I am using the Aggregator to group the areas, and that is looking fine in the Inspector.When I come to write these features to an esri ArcGIS Portal Feature Service, I receive the following error:<type 'fmeobjects.FMEAggregate'> cannot be converted to esriGeometryPolygonI am building Homogeneous Collections where possible, but even the admin areas that are made up of a single contiguous area have "fme_aggregate" as "the fme_geometry(string)" attribute in the inspector.

21810

7 months ago

jasmus2Contributor

asked in Data

Bentley .cit files and nm1 files

I am trying to bring in some files that are dgn/bentley related (cit and nm1) and I can’t figure out how to read these files or view them. I found in a search this site that says how to read a .cit file but I can’t find this or how to view these?https://docs.safe.com/fme/html/FME-Form-Documentation/FME-ReadersWriters/ingr/ingr.htmI do not see this INGR in my lists or anything when I put in reader format .cit

franco69Contributor

Hi all,i have following problemi join a gdb (with included domains) with an excel sheet…...if i don’t resolve the domains the feature class is written but no geometryif i resolve the domains i have geometries right but no domains anymore Can somebody help me? Thx Franco

391

7 months ago

hawaiialexContributor

asked in Data

Resolve Domains Replication of FCs in EGDB

Hi all,I am working on a script to replicate a set of feature classes from one enterprise geodatabase that we use for transactional editing purposes to another enterprise geodatabase that we use for publication purposes for the organization. This replication script would be scheduled in FME Server to run on a nightly basis.The issue is that the feature classes I plan on replicating have several domains and subtypes applied to them that don’t translate correctly.I’ve been advised to use the ‘Resolve Domains’ functionality and change the configuration from ‘coded domain’ type to ‘char’.My test workflow involves the below:Copying the feature class via the ‘Import Feature Class’ tool in ArcGIS Pro into the target Enterprise Geodatabase to have the feature class available in both geodatabases With FME:Selecting the ‘Resolve Domains’ checkbox in the Writer at the outset, and using the ‘Insert’ and ‘Truncate Existing’In the Writer configuration, changing the ‘coded domain’ type to ‘char’ and

450

7 months ago

spill002Contributor

asked in Data

Request for Inputs - Rerun Entire Workspace in FME Server

Hello,I have a workbench, which when published to FME Server - is not updating the data, rather - I am getting the cached data. Is there any way that when I publish the workbench to FME server- can I use the option of “Rerun Entire Workspace” rather than “Run Workspace”. Please find the below snapshot of the options in workbench : Also, want to mention that - when I did ran the workbench in FME Desktop - it updates the data only when I use the option “Rerun Entire Workspace”, but when I use the option “Run Workspace” - I am seeing the cached data. Please let me know If I am missing anything.

503

7 months ago

parke372Contributor

asked in Data

Replace Line Feeds/Carriage Returns and Tabs with properly recognized JSON

Hi all,i’m building an FME desktop workspace for eventual use with a server automation. we’re taking some emails, parsing them for some specific data, reading text files with additional information, and then putting it all together to be sent out via an API for a software called Desk Alerts.Where i am running into issues is - the text files are UTF-8 encoded and contain line feeds and tabs that the API isn’t happy with.so for instance i might have a message that looks like: now you can see i’ve got 4 tabs and almost 20 line feeds.this text is actually the combined result of several files and the content is variable so for readability’s sake maintaining the line feeds (and preferably the tabs) is paramount.When sending to the API, i get the following error response: {"status":"Error","errorMessage":"Request model invalid","data":{"validationErrors":["[queryParam] : The queryParam field is required.","[$.body] : '0x0A' is invalid within a JSON string. The string should be correctly escap

2722

7 months ago

mukhtargContributor

asked in Data

Sanpping Polygon to Polygon

Hello Experts, I have two Layers PD and AA. I want to align the PD layer to AA layer. Any help would be highly appreciated. Thanks in advance

823

7 months ago

outlookContributor

asked in Data

st_geometry to (esri shape column) p

Hi All, I want to update geometry column (esri SDE) from postgresql writer. What I need to do ? esri keeps the point value like. 2D000000010000000100040001000000110000000101000090D1D3C2A00380A6E4AFBA0880A8D6B90700000000 I can reading from the postgresql as sde.asText / asBinary but I can not write

706

7 months ago

Page 19 / 272

Badge winners

alkamelgishas earned the badge Welcome Back
alkamelgishas earned the badge Integrate Data with the FME Platform Training
valentin98has earned the badge FME Form Basic Training
marjan_nikoukarhas earned the badge Student (Bronze)
grahamavardhas earned the badge Welcome Back

Show all badges

30,126: Posts

Terms & Conditions Cookie settings

Sign up

Already have an account? Login

An FME Account is required to contribute

Login to the community

An FME Account is required to contribute

Enter your E-mail address. We'll send you an e-mail with instructions to reset your password.

Enter your e-mail address

Back to overview

Scanning file for viruses.

Sorry, we're still checking this file's contents to make sure it's safe to download. Please try again in a few minutes.

This file cannot be downloaded

Sorry, our virus scanner detected that this file isn't safe to download.

A product of

Company Products Giving Back Careers Contact Privacy Feedback

Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.