Skip to main content
  • EN

Data

Using readers and writers to extract and load data.

  • 7,092 Posts
  • 26,104 Replies
7092 Posts
adam_timmons
Observer
adam_timmonsObserver
asked in Data
Bentley Microstation Reader Not Showing Referenced Files

Note: we are primarily using FME Form version 2023.2.4.0 (build 23785), but have also tested this on 2024.0.1.0 (Build 24202) with similar results.We have received DGN files that are referencing models that are both internal and external to DGNs themselves. These external DGN files are located in the same directory folder as the file being read. We are interested in having the reader (Bentley MicroStation Design (V8)) read not just the primary DGN, but also the referenced models.In the parameters for the reader, we check the box for “Read Elements from Reference Files” and set the value for “Max Depth of Reference Files to Read” to something arbitrarily large like 20.Selected parameters, but have tried a variety of combinations of values with no difference in outcomes.Because we are interested in reading data from different files, the “Single Merged Feature Type” option is selected.When the reader is run, we see data from the primary DGN, but nothing from any other DGN that it it is re

1291
saraatsafe
Safer
gogopotter90
Contributor
gogopotter90Contributor
asked in Data
How to use Python script into FME ?

Hello , i have a Python script works good into Pycharm without problems , to do some tasks on excel file then i am using the new excel file output of Python as input in FME .i would like to use Python caller , to put the same script inside FME.how could I change the script to be compatible with Python caller ?which changes should I consider and think about it  ?Thanks in advance 

1592
gogopotter90
Contributor
emerson_styles
Contributor
emerson_stylesContributor
asked in Data
How to add nodes on a line with point data

I’m doing a routing analysis and I have a road network that I need to create new nodes along where I have points for addresses. I used the AnchoredSnapper transformer to ensure that the points align with the line data. The point data (addresses) is a separate dataset from the line data (network). Does anyone know how to create new nodes along an existing line using point data?

682
liamfez
Influencer
fmelizard
Safer
fmelizardSafer
asked in Data
Proper use of the OSM reader?

I'm trying to load OpenStreetMap using FME. This works great with small datasets in 2012, however as soon as I start playing with large datasets (millions of features) I hit the problem that the FME OSM reader acts as a grouper - it reads the entire dataset in before it actually trys to translate everything.  Does this have to happen? Is there a way to disable it or is this a limitation of the XML reader? It's massively slowing down everything because even with 32GB of RAM, FME is still filling it up and then writing to temp disk space (despite the original dataset only being about 7GB uncompressed).  Using 2012 SP4 64bit. 2013 13206 is 5.5 times slower on the small dataset, so not using that despite changes in #13082 to make "it better with large datasets" - not sure what that means.

416
dustin
Influencer
nanchuangyeyu
Contributor
nanchuangyeyuContributor
asked in Data
How to merge muliple Personal Geodatabases(mdb) with a selected mdb as schema template?

Hi everyone, I have several Esri personal Geodatabases with similar schema but also some tiny differences.That is, they are required to have identical schema but in fact some don't due to various reasons. Some feature types and attributes are unexpected, while some required ones are missing. Now I want to choose a standard one from them as a template and merge them all to get a standard destination mdb. How can I achieve that?

361
liamfez
Influencer
harmen
Contributor
harmenContributor
asked in Data
User parameter: selecting table from file geodatabase

Within the User Parameters I have added the option to Select a File.More specific the user has the option to select a Geodatabase (Items to Select).When the user has selected the Geodatabase, I like to have the user also select the database of his/her own choice within that Geodatabase. Is there a way to configure that?  

632
nielsgerrits
VIP
rudy_v
Contributor
rudy_vContributor
asked in Data
ESRI Geodatabase error in Dynamic Writer - Feature Type Definition not defined.

 Hi,I am just in the process of migration testing from FME Flow/Form 2023.1 to 2024.1. It seems all of our GeoDB dynamic writer fails the following error:Esri Geodatabase Writer: Feature Type Definition 'd_y_n' was requested, but not defined. Please visit https://www.safe.com/support All existing runs perfect on 2023.1 Flow and Form - not sure why it fails. Tried all possible scenarios, still fails. I have changed it to a normal writer - then it worked fine.GeoDatabase - SQL Server 2017, SDE Version 10.9.1 (pro 2.9 created)Pro Version 3.3.1 installed on PC / ServerRudy

4132
rudy_v
Contributor
O
omeaga3son
asked in Data
write a list of gdb feature classes to excel

Hi,  I am a total novice with FME - will be going on some training soon, however I am trying to get familiar with the software prior to that.  I have what I think is a very simple task but can't seem to figure it out.  All I want to do is create an excel file with all of the feature classes contained in my geodatabase.  My geodatabase (version 10.1 in ArcGIS) contains feature datasets - although I do not these listed in my excel output.  Sorry for asking such a basic question - I am new to the knowledge areas/and community

24314
keith.addison
Contributor
ed-progres
Contributor
ed-progresContributor
asked in Data
Can FME read in data from Oracle SQL Views?

Hello,I was wondering if FME can read in  data from Oracle SQL Views?If yes, great. How?

763
ed-progres
Contributor
ricardoa
Contributor
ricardoaContributor
asked in Data
Create polygons from a string of coordinates

Hi there!I want to create polygons from an attribute that contains a string of coordinates and other information. My problem is that I don't know how to only extract the coordinates from the string and then create the polygons by their ID attribute.  The string in the coordinates attribute looks like this:“{"type":"Feature","properties":{},"geometry":{"type":"Polygon","coordinates":[[[13.707920553133594,60.03475791657763],[13.708950521395312,60.03304306231432],[13.705474378512012,60.03160680337132],[13.702513219759567,60.02997753853561],[13.702255727694139,60.030577803349054],[13.703929426119434,60.03184261136259],[13.706161024019824,60.032571461785544],[13.707920553133594,60.03475791657763]]]}}” ID attribute 1 to 10 Regards,  Ricardo :) 

996
nielsgerrits
VIP
ricardohhd
Participant
ricardohhdParticipant
asked in Data
How do I keep the attributes from a DXF?

I have a DXF file that consists of multiple geometries and attributes. I want to convert this DXF file to a file geodatabase with multiple layers based on each geometry type in the DXF. The different layers also need to retain the same attributes as the DXF files. I attached the file as a zip to this post.I tried loading the DXF file in FME using the Autodesk Autocad DWG/DXF reader and splitting the geometries using a GeometryFilter. That worked fine. However, the attributes disappear when I read the DXF file in FME. Do you have any suggestions on how to retain the attributes of a DXF file when reading it in FME and converting it to a GDB? The data was gathered with a Leica gs18T in case that helps.

671
thomasmarti
Contributor
alicevdb
Contributor
alicevdbContributor
asked in Data
WHERE Clause can read text with macronated o, but fails to read a User Parameter with same text macronated o.

Context: Feature Type with a WHERE clausePurpose: To only read records relating to a user defined Iwi (Maori Tribe).When the WHERE Clause is, for example: "IWI"='Whakatōhea', the transaction is successful and returns the expected records. However, when the WHERE Clause is set with a User Parameter "IWI"='$(IWI)'and the User Parameter is set with the value Whakatōhea, the transaction is successful but no records are found.This behavior is only affecting records with macronated vowels, which is common in Maori language.  Users need to be able to input their Iwi of interest for this workbench.Can anyone suggest where the WHERE clause setup might fail?Many thanks  

273
alicevdb
Contributor
mr_fme
Enthusiast
mr_fmeEnthusiast
asked in Data
Problems with feature count server

Hi,Suddenly, the feature count in FME Desktop disappeared, even with the "Enable Feature Counting" option ON.The log shows: Cannot connect to feature count server "65051."Why does this happen?The problem occurs with all formats.Thanks.

412
mr_fme
Enthusiast
F
freegnome
asked in Data
Unable to convert value taken from Z_ORIGIN keyword to a float data type

Hello,I am writing to ESRI geodatabase that uses MS SQL. Every time I get "Unable to convert value taken from Z_ORIGIN keyword to a float data type". So far i have:Used a 2dforcer,changed the "z scale and z origin" to 0 or 0.00000under table creation parameters set "Contains Z Values" to No (also set to yes, and blank on other trys)Recreated the writersSet the writer geometry to table, not a point file.Nothing has worked. I am under the assumption that the writer is passing a "Null" value to the writer which in turn says that "null" cant be a float value.Finally I have looked at the other posts on this and nothing has helped. Any suggestions? Just to update, I used a vertex creator to make new geometry, and still didn't work, but i can insert all of this into a file geodatabase and then import that to my enterprise database. If anyone has any help please post it, this issue doesn't have a good answer on the forms yet.

17111
arunkj
Contributor
tbarker1986
Contributor
tbarker1986Contributor
asked in Data
DWG Writer: Preserve Template File Paper Space Issue

Hi All, i’m having an issue writing to an AutoCAD DWG file, with a template file. I want to preserve the Model Space, but not the Paper space as i’m editing some attributes within the drawing title block in FME which I want to write back out in the paper space.Even though I am setting preserve paper space in the writer to No, the original blocks from the template file, including the original attribute values, are being written as well as a copy of the block with the new attribute values I have updated in FME. As a result I have 2 overlapping blocks, 1 with the new values I want to keep and 1 with the original values I am trying to replace. I just want a single block with my new values.In the attached, you can see the output from the writer, with the same block duplicated with the new value I have updated and the old value in the template (highlighted in yellow).Any thoughts/ideas/assistance with this would be greatly appreciated, whether this is possible or not possible. I have done si

591
fornberry
Participant
lambertus
Enthusiast
lambertusEnthusiast
posted in Data
PostGIS performance tricks

Recently I put different tables into a PostGIS database. One of these tables contains 23M unique records and does not contain spatial data. I would like to read them into a FME workbench. Do you have some suggestions how I could improve my reading process?Is it possible for example to preselect already a set of columns I need while reading my table using 'SQL to Run before Read'?’ If so do you have a example SQL querry?   Is there a performance difference between  'SQL to Run before Read'?’ and the WHERE clause option in the Constraints menu? 

762
lambertus
Enthusiast
mr_fme
Enthusiast
mr_fmeEnthusiast
asked in Data
Run two custom transformer identical in Workspace

Hi,I'm using two SentinelDownloaders in my workspace: one to download data from a specific year and another to download data from the previous year. At the end, my workspace shows a "FAILED" message in the API after completing the first SentinelDownloader. I understand there is a conflict because it's the same transformer being used for different dates in the same execution.How can I run this workspace without problems?Thank´s

171
ebygomm
Influencer
romar
Contributor
romarContributor
asked in Data
Oracle connection timeout for long running workspace

Hi! I have a workspace with a runtime of effectively 28 hours (this workspace controls slave workspaces, that get 50 million rows of external data). The runtime is only this long for the initial running of the workspace, However it is possible to get severe peaks in runtime due to hystorical data recalculations. I finish the workspace with a SQL executor that merges the retrieved data with already existing data and does some other cleanup. At this point this transformer fails though, forcing me to run the queries manually. I have not saved the log due to my brilliance :). the error given is paraphrased: there is no connection with the oracle server. I think the issue is that a connection is established previously with the oracle server, than put on hold for about 28 hours, than necessary again, with an elapsed acces key. Is there a way to circumvent this? I have not tried, due to time commitment, to untick persistent connection, should that work?

621
hkingsbury
Celebrity
nea
Contributor
neaContributor
asked in Data
Not able to create a web connection

i”m unable to create the web connection despite all the documentation.  

849
nielsgerrits
VIP
topotoma
Contributor
topotomaContributor
asked in Data
Featurewriter Excel error "Named range from /xl/workbook.xml part" only with 2024.1.2.0

Hello,When using Featurewriter with a template excel file, i encounter en error when trying to open the written excel file which indicates a problem with the Named range "Named range from /xl/workbook.xml part".This happens only since updating to 2024.1.2.0, if I use the same workspace with 2022 version it’s fine.Has this happened to someone else ? is there a workaround ?I’ve tried a lot of things but couldn’t find the issue so for now we run this workspace with 2022.Thanks for your help.

510
topotoma
Contributor
pushpa
Contributor
pushpaContributor
asked in Data
Column names on Oracle VIEW doesn't show on FME workbench.

Hi there,I have placed few VIEWS on Oracle Database changing TABLE column names. ie: ID as PLANT_ITEM_IDWhen I connect the VIEW via FME 2023.2.1.0 all column names are showing as what is recorded on Oracle Tables. It is ignoring column names I have set up on VIEWs. How do I get around this issue?I don’t want to use “attribute manager” to change the names. I have already done that on my VIEWs.Is there a way I can see the Column names on VIEWS through FME?thanksRegards,Pushpa  

441
kailinatsafe
Safer
mr_fme
Enthusiast
mr_fmeEnthusiast
asked in Data
Extract pixel

Hi,I need to extract pixel with value 3 ,8 and 9. Have some transformer that filter the pixel value? Thank´s

21325
jkr_wrk
Influencer
dustin
Influencer
dustinInfluencer
asked in Data
Help with understading OSM donuts

I could use some help in understanding how donuts are stored in OSM XML. Or more specifically, why FME Data Inspector is not constructing the donut from the XML. Here is the simple shapefile test data I’ve created.In FME, I’m writing out the OSM using the normal writer:When I read that OSM back into Data Inspector, it doesn’t recognize the shape as a multipolygon. The geometry type is fme_no_geom:The XML code is below. I’m seeing the relation present, so I’m not sure what else is required.<?xml version="1.0" encoding="UTF-8"?><osm version="0.6"> <node lat="34.6596384" lon="-87.0320968" id="-7"/> <node lat="34.5631538" lon="-86.8849822" id="-10"/> <node lat="34.7427899" lon="-86.9769288" id="-8"/> <node lat="34.4741373" lon="-86.9217608" id="-4"/> <node lat="34.6293808" lon="-86.7401662" id="-2"/> <node lat="34.5063457" lon="-87.0619795" id="-5"/> <node lat="34.5536885" lon="-86.9194622" id="-16"/> <node lat="34.5669396" lon="-87.

881
emma.at.safe
Safer
smithgk
Contributor
smithgkContributor
asked in Data
Converting Overpass Turbo (OSM) data from xml

Hello,There have been some previous posts about converting OSM data in FME, but I’m hoping to breath some new life into the conversation. I have an Overpass Turbo query here meant to query ski area boundaries, lifts, and runs within Colorado. The query works just fine in Overpass Turbo, and I’m able to read the raw XML data into FME with the HTTP Caller no problem. The issue I’m running into is visualizing the data beyond that. I’m aware of the example workflow that is similar to my process, but I don’t understand how the XSLT Processor and XML Fragmenter transformers work. If I use those transformers in my workflow, all the coordinate information seems to disappear when the data is fragmented.Does anyone know how to properly display the XML data in FME? I might need a geometry filter to split out points, lines, and polygons, but I don’t think that’s what is holding me back currently. Any help would be greatly appreciated!

1171
todd_davis
Influencer
scarecrow
Contributor
scarecrowContributor
asked in Data
Raster renaming in Esri SDE

Hi All,I’m creating an archiving workbench for our Esri Enterprise SDE databases. I’m trying to automate checking for an existing version of a raster in either our shared or archived databases and then assigning a version suffix according to if the data already exists in one of the databases. I have the majority of the workbench sorted but struggling to automate renaming the raster to include the version suffix when writing to archive.In my example below I’ve identified that there are two existing versions of the test data “gen_aaa_aaa_test_ras” so I’ve created a name field “archhiveddataname”, which contains the the name I want to give the raster when I write it to Archived “gen_aaa_aaa_test_ras_2”. I can access this information from the schema port but not from the generic. If I include the information from the schema port I get an error saying that I’m trying to write non-raster geometry and the workbench fails. I’m using FME 2023.1.2.0 (20231110 - Build 23641 - WIN64) Thanks in adv

221
hkingsbury
Celebrity

Badge Winners

Show all badges

Community Stats

31,807
Posts
120,824
Replies
39,436
Members

Latest FME

Terms & ConditionsCookie settings
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.