Skip to main content

Data

Using readers and writers to extract and load data.

  • 6,964 Posts
  • 25,642 Replies
6964 Posts
ifawzi
Participant
ifawziParticipant
asked in Data
Reading CSV files with different schemas and writing to SQL server.

Hi, I am new to FME, and I am trying to build an FME workspace that reads multiple .csv files from a folder and write each file to separate table in SQL server. Each CSV file: has different schema should became it is own SQL table Here is what I tried so far: I used Directory and File Pathnames reader I used Feature Reader The <Generic> port of Feature Reader shows no data, I am assuming because the schema of each is different. the <initiator> port give me the correct file name and path.

192
crutledge
Enthusiast
dsign-world
Contributor
dsign-worldContributor
asked in Data
What are valid values for a custom schema's data types in a dynamic wokspace?

The website Dynamic Workflows: Destination Schema is Derived from a Lookup Table > Creating a Table-Based Schema > Geometry Type tells me that valid values for the `geometry_type` column are the values from the `fme_type` format attribute. It provides the full list of values under this link. The link is incorrect. This is the list of FME internal data types. These are NOT valid values from the `fme_type` format attribute and these are NOT values from the `geometry_type` column in the screenshot above in the tutorial. Where is the correct list of values for the `geometry_type` column or the `fme_type` attribute?

351
AliAtSafe
Safer
joku041199
Contributor
joku041199Contributor
asked in Data
Reading an Excel file exposing all available columns automatically (To create Web App)

Hey Everybody, In my work, I generally get the question from clients to build a FME Server Web Application. Often, the question involves a MS Excel file, which needs to be analyses and converted to points/lines/polygons (GIS). Mostly, I use a Creator to kick-off the Workspace and a FeatureReader to load in the Excel. Also, I force the FeatureReader to have one single output port (Generic) and load in the Excel file using a User Parameter when kicking off the Workspace. This means that I need to Expose all attributes individually in the Workspace. However, this limits the flexibility. When I want to create a Web App from the Workspace, the input Excel always needs to have the same columns (so a Template Excel as input). An potential extra column is therefore not exposed an not in the result of the Workspace. However, in some cases, it is for my clients not possible to always use an Excel Template. Or they want to add additional information. My question therefore is, is it possible to dy

395
crutledge
Enthusiast
simeon.simic
Contributor
simeon.simicContributor
asked in Data
dwg coordinate system

I am reading dwg file, but i cant set him the right coordinate systemThe file i am reading is from autocad 2012In autoCad coordinate system is set to EPSG : 31277, i dont have that system in my fme workbench Thank you in advanced

80
simeon.simic
Contributor
fionasin
Contributor
fionasinContributor
asked in Data
Method for forcing decimal points between numeric characters in a string?

I have various values in a 'code' field. A sample would be: A33, B111W, B12, C3V etc, etc. This issue is, there should be decimal points between all numbers, i.e, these should correctly read: A3.3, B1.1.1W, B1.2, C3V. There does not need to be points between the letters. I can string replace “33” with “3.3” and continue to do that for all values if I have to, but I’m hoping that there is a more efficient way of capturing these values and using regex or something to force decimal points between the numerical characters. I’m not well adept at using the stringformatter, but maybe there is something there?

282
fionasin
Contributor
caracadrian
Contributor
caracadrianContributor
asked in Data
Autodesk AutoCAD DWG/DXF repairing polyline on read

I am trying to create a workspace that identifies consecutive duplicate point in DWG polylines. and outputs their coordinates for future review. As per my example, Autodesk AutoCAD DWG/DXF (normal reader or FeatureReader) does some autorepair on the features while reading. The DWG feature has 12 vertex in AutoCAD but only 7 in FME. Please see attached sample feature in dwg format. I want FME to read the same geometrie that AutoCAD gives. Using FME 2024.2.1.0 build 24801

413
caracadrian
Contributor
emerson_styles
Contributor
emerson_stylesContributor
asked in Data
Output excel file hiding column names (row 1)

I am writing data to an excel file but when I open the written file, the first row of the file (that includes the column headings) is hidden and I have to manually expand it to show the headings. Is there a way to fix this in the writer at all?

7679
phoeffler
Contributor
brendanneal
Contributor
brendannealContributor
asked in Data
Snowflake database connection with keypair not recognising schema

Hi, I have and issue trying to connect to snowflake DB using keypair authentication, it is not recognising my schema. I'm using FME(R) 2024.1.2.1 (20240906 - Build 24624 - WIN64). I can connect fine in DBeaver using the keypairs. I'm using the advance JDBC settings to add the parameters in a new snowflake database connection. The error I get is ' C OM.safe.fme.jdbc.TranslationFailureException: java.lang.IllegalStateException: Connection established with no current schema. Make sure that schema 'dbo' specified in connection settings exists and is accessible by the user ‘. I know you need to have the case correct in the parameter I have checked with the DBA and have been told that the schema is definitely lowercase. I can’t show all the jdbc connection string, but this is the schema bit ‘ schema=dbo ’ so it is passing the lowercase parameter. Again, I am using the lowercase in DBeaver and connecting fine. I’ve also tried the scheme DBO, “dbo”, “DBO”. There is another schema in the databa

324
brendanneal
Contributor
bruno
Contributor
brunoContributor
asked in Data
IfcZone writing

Hello to all,3 months ago, kailin told me that the 2024 version did not allow writing zones. I would like to know if it is the case now. If not, what is the planned schedule because for the moment we are blocked for the enrichment of our models Best regards Bruno

232
bruno
Contributor
aawad.93
Contributor
aawad.93
asked in Data
ChangeDetector Still Flagging Geometry Changes Even with Lenient Matching and Vector Tolerance

Hello everyone, I’m working in FME 2023.2 and using the ChangeDetector to compare features coming from PostGIS and ArcGIS Online (AGOL). Visually, the geometries look identical (no edits were actually made), but the ChangeDetector keeps reporting that a large number of features have updated geometry. I’ve already tried: Turning on Lenient Geometry Matching in the ChangeDetector Setting Vector Tolerance to a small value (like 0.0001 - Takes more time) While that helped reduce the count somewhat, I’m still seeing a substantial amount of features flagged as updated when they shouldn’t be. I suspect it might have to do with differences in ring orientation, vertex ordering, or the presence of Z/M values, but I’m not certain how best to address these discrepancies so the ChangeDetector recognizes that there hasn’t really been a change. Any suggestions or advice on how to make the geometry truly match (or be treated as unchanged) would be greatly appreciated. Thanks in advance!

684
crystalatsafe
Safer
tnarladni
Enthusiast
tnarladniEnthusiast
asked in Data
Date written as text in excel

The process is to read from a sql database, write to excel file, then import that excel into another software. The software expect one of the fields to be a date field. The data coming out of the database is datetime. I used the DateTimeConverter to go from 20190101000000 to 01/01/2019. I'm using an excel template so I'm setting that field to be of type date. The writer in the workspace is set to be of type date. But the data that is written is text. I can tell it is text because it is left justified instead of right. If I click into the cell and press enter, the data is then right-justified and I'm able to import the excel into the software. I've looked up this issue and the most common solution I found is to do a "Text to Columns" under the Data tab in excel. I did this in the template, but the data written is still a text I thought maybe it was the DateTimeConverter transformer, so I created a view in the database to make it a date instead of a datetime. It is still writing out as a

45220
crystalatsafe
Safer
tim.albert.vaa
Contributor
tim.albert.vaaContributor
asked in Data
Format Text File where Column Names are Repeated on Each Row

I have a text file that looks like this: "CommonCode":"EI","IATA":"EI","ICAO":"EIN","CarrierName":"Aer Lingus""CommonCode":"AM","IATA":"AM","ICAO":"AMX","CarrierName":"Aeromexico" I want it to look like this: CommonCode IATA ICAO CarrierName EI EI EIN Aer Lingus AM AM AMX Aeromexico I want it to create the column names dynamically from the text file as the column names will change and increase or decrease in number. In R there appears to be a command called: “unsparsify” Which according to this post seems to do it.https://datascience.stackexchange.com/questions/55879/file-format-where-column-names-are-repeated-on-each-row I can do it using a combination of AttributeSplitters, ListExploders, ListIndexers, AttributePivoters and AttributeExposers. But I can’t seem to get away from having to Explicitly input the Attribute names in the AttributeExposer and I need it to be dynamic. Any ideas?

827
ebygomm
Influencer
andreaw
Contributor
andreawContributor
asked in Data
Missing OSM lines with OSMPBF reader

When i import OSM.pbf data in FME with the OSMPBF reader the following warning pops up: During translation, some features were read that did not match a reader feature type in the workspace. This can happen if the reader dataset is changed, or a reader feature type removed or renamed. When i open the data in QGIS, the missing lines are there. Version 2024.2

383
kailinatsafe
Safer
rmpayaby
Contributor
rmpayabyContributor
asked in Data
Reading and IFC-file, reproject it and write back to new IFC-file

I am reading an IFC-file using a FeatureReader with the following configurations: I am using the <Generic> port to connect it to a Reprojector-transformer. After reprojecting it, I want to use a FeatureWriter and write it as a new IFC-file. I am however stuck on how I should set up the FeatureWriter. I have managed to get a new IFC-file, but the file size is quite small. I am interested in automating the process, and want the FeatureReader to accept any types of IFC-files. Any tips on how I can solve this?

461
daveatsafe
Safer
vdeyepc
Contributor
vdeyepcContributor
asked in Data
MapInfo transformation not reading right Coordinate System

Hello FME Folks,This is related to MapInfo Tab files. Since we upgraded from FME 2018 to FME 2024.2.1. The MapInfo coordinate System (Non Earth Meters) are not detected which used to happen in FME 2018. Instead it is taking wkt_auto_1 and is misaligned. Please let me know what changed.

386
daveatsafe
Safer
pragathjobs
Participant
pragathjobsParticipant
asked in Data
LineOnLineOverlayer after upgrade from FME 2020 to FME 2024 output count difference

LineOnLineOverlayer after upgrade from FME 2020 to FME 2024 output there is a count difference in outputs. The data used for both version is same, In FME 2020 the tolerance parameter was none while there is no such option in 2024, so used 0 as tolerance value. Still showing count difference.

292
philipwhitten
Contributor
tauseef144
Contributor
tauseef144Contributor
asked in Data
Assistance needed with comparing location across two Datasets :

Dear All,I’m working with two datasets that share a common key: Station ID. Each dataset provides coordinates ( latitude and longitude) for these stations. My objective is to determine whether the station with the same ID have identical location in both datasets or if there are discrepancies. To achieve this, I need to calculate distance between the corresponding coordinates from both datasets. While this seems straightforward, but I encountering challenges in implementing it effectively. I would appreciate any help on the best approach. Thank you in advance. BR//

373
tauseef144
Contributor
edwibberley
Contributor
edwibberleyContributor
asked in Data
ESRI ArcGIS Server Feature Service reader won't read feature type. Error - ArcGIS Feature Service Reader: Encountered an unexpected error. The error code from the server was '500' and the message was: 'Error performing query operation'. Details: ''

Hi, I have a workspace using an ESRI ArcGIS Server Feature Service reader with a token authentication. I am trying to read 3 feature types from the map server and 2 of them read fine but one comes back with the error: "ArcGIS Feature Service Reader: Encountered an unexpected error. The error code from the server was '500' and the message was: 'Error performing query operation'. Details: ''" Does anyone have any ideas of why only one of the feature types would not work? The three feature services should all have the same naming standards and schema so there is no reason why they should behave differently that I can see. Are there any suggestions for how I might be able to find the source of this issue? Any help would be greatly appreciated! Thanks.

11718
matthewb
Contributor
simeon.simic
Contributor
simeon.simicContributor
asked in Data
CSV parameters reader

I have network based resource folder, that folder si on fme flow When file arrive in that folder, i need to write that file on other location, i just need help i don t know what parameter or what pointer should i send to my reader that it knows which file should write

505
j.botterill
Influencer
E
evangoodwin
asked in Data
SDE (Sql Server) Truncate Issues - FME fails to truncate table

Hi, I have a client that is experiencing some odd behavior when it comes to truncating feature classes and tables in a SQL Server SDE environment. The writer is set to 'Yes' for Truncate table however when fme writes out the features heading to the writer it does not truncate the data first but merely inserts data records. If i add a new reader, move the feature types over, configure it exactly the same and rerun the writer functions as expected (Truncate and Load data). Why do i have to add a new reader that is configured exactly the same to get FME to truncate properly. Do FME writers go stale? FME 2015.1 SDE (10.4.1) Any help will be apprecaited.

633
evieatsafe
Safer
pkno
Contributor
pknoContributor
asked in Data
How to: Feature based Filter?

Hello everyone, I was wondering if there is a simple way to filter one Dataset by comparing key-values with another Dataset. Basically a Join but onlky keeping Left Attributes. Example: Dataset1 Project SubID moreAttributes from Dataset1 ABC 1 ... ABC 2 ... DE 1 ... DE 2 ... Dataset 2: Project SubID different Attributes from Dataset2 ABC 1 ... ABC 1 ... ABC 3 ... ABC 3 ... DE 2 ... FG 3 ... I want to Filter Dataset 2 to only keep the ABC|1 and DE|2 rows because they match dataset 1. I am sure I could accomplish this via join and subsequently throw out all the Attributes that were introduced from dataset 1, but I was wondering if there is an easier way. Thanks all

284
ebygomm
Influencer
brandonguo
Contributor
brandonguoContributor
asked in Data
FMW Files Disappearing After Closing Workbench

This is really strange—I’ve never seen this behavior before. After saving and closing my workbench, all other FMW files in the same folder disappeared automatically. At first, I thought I was imagining things, but other users have reported similar issues before. [Post link] Version: 2024.2.1 (Build 24801) OS: Windows 11 24H2 (26100.2894) The only change I can recall is that I recently uninstalled all old FME versions, but that shouldn’t somehow trigger file deletion, right?

889
featuremichael
Supporter
pragathjobs
Participant
pragathjobsParticipant
asked in Data
Difference in feature count of Neighborhood Finder transformer after upgrading from FME 2020 to 2024

Difference in feature count of Neighborhood Finder transformer after upgrading from FME 2020 to 2024. The count differs even though the input data and projection are same. Similarly for lineonlineoverlay transformer the tolerance 0 and automatic gives the same output while the expected was different as FME 2024 does not have an option to give “none” in tolerance.

271
j.botterill
Influencer
jiri.petrak
Contributor
jiri.petrakContributor
asked in Data
DGNV8 writer throws AutoCAD2004Writer error

Hi, I am converting SHP data to DGN using DGNV8 writer dynamically (via FeatureWriter). Everything works fine, but I don´t understand why it always uses ACAD writer first, throws this error and warning: AutoCAD Writer: An error occurred for dataset 'dgn\DI\vydej_di_L1.dgn', in function AutoCAD2004Writer::write. Unknown exception thrown. Aborting methodBADNEWS: ACAD writer: An error has occurred. Check the logfile above for details(D:\code\fme\source\engine\engine_api_impl\cpp\pluginbuilder\dynawch.cpp:129) class StatusInfo __cdecl DynaWriteChannel::putFeature(class STFFeature *&) DESIGN WRITER: Failed to write '3D elements' data through a proxy. Skipping such data for all elements written Then it writes everything using DGNV8 writer and the output DGN looks good, nothing seems to be missing. The only issue is that it throws quite big number of errors like this. I publish and use this workbench in FME Flow and the number of errors doesn´t look good in the finished jobs. Any idea how to

695
emmaatsafe
Safer
nordpil
Supporter
nordpilSupporter
asked in Data
Bug in JDBC SQL Server Spatial Writer - when geometry is empty

Hi, I have a very basic workbench in FME 2022.2.8 Reading features from a MSSQL table and writing to another, it is a FeatureReader/FeatureWriter combon in a dynamic workflow using the schema from the FeatureReader. This is using JDBC MSSQL Spatial. If the geometry column is set to EMPTY the writing fails with the message: MS SQL Server Spatial (JDBC) Writer: Error writing feature. Error: IFMEFeature.exportGeometryToOGCWKB: Unable to export geometry to OGC WKB Examining the feature in the database one can find it using SHAPE.STIsEmpty() = 1 and SHAPE.STAsText() returns “GEOMETRYCOLLECTION EMPTY”

613
hkingsbury
Celebrity

Helpful Members This Week

Badge Winners

  • FME Form Basic Training
    siyuliworkhas earned the badge FME Form Basic Training
  • Welcome Back
    jjfhas earned the badge Welcome Back
  • Teacher (Bronze)
    arenscotthas earned the badge Teacher (Bronze)
  • Teacher (Grey)
    jmfeehas earned the badge Teacher (Grey)
  • Socializer (Ice Breaker)
    jmfeehas earned the badge Socializer (Ice Breaker)
Show all badges

Popular Tags

Community Stats

31,082
Posts
117,847
Replies
38,837
Members

Latest FME

Terms & ConditionsCookie settings

Cookie policy

We use cookies to enhance and personalize your experience. If you accept you agree to our full cookie policy. Learn more about our cookies.

 
Cookie settings
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.