Skip to main content
  • EN

Data

Using readers and writers to extract and load data.

  • 7,029 Posts
  • 25,888 Replies
7029 Posts
dcox
Contributor
dcoxContributor
asked in Data
Hel with writing to Excel Template

My organization has a platform that allows users to collect spatial data. We have a forest plot survey that allows users to collect information about forest characteristics. The form is intense, with many nested layers, many-to-one relationships, and repeating sections. I have developed a workbench that reads the JSON from our organization website and ‘unpacks’ it into it’s individual tables, all with id’s to perform the necessary joins down the road. Previously I was just writing all the tables to their own sheet in an excel document, but now my managers are asking me to populate the data into their preferred format, so they have given me an excel table to show what they want the output to look like. It’s not a data friendly format. Essentially, they want all the data to populate onto a single sheet where the sheet has multiple instances of different fields names in the same column. Think like multiple tables stacked on top of each other in one sheet.  My initial idea was to write all

331
ebygomm
Influencer
heidivanparys
Participant
heidivanparysParticipant
asked in Data
Create GML file with a feature collection in a custom namespace, including a bounding box

I am trying to create a GML file that complies to the GML simple features profile (GML-SF) (http://portal.opengeospatial.org/files/?artifact_id=42729). I already created a GML application schema that complies to the GML simple features profile. It follows the pattern described in GML-SF, 8.4.2:<schema xmlns="http://www.w3.org/2001/XMLSchema" xmlns:ex="urn:uuid:3c615c60-013b-4006-bf53-d0ba4fb5fc94" xmlns:gml="http://www.opengis.net/gml/3.2" elementFormDefault="qualified" targetNamespace="urn:uuid:3c615c60-013b-4006-bf53-d0ba4fb5fc94" version="1.0.0"> <import namespace="http://www.opengis.net/gml/3.2" schemaLocation="http://schemas.opengis.net/gml/3.2.1/gml.xsd"/> <element name="FeatureCollection" type="ex:FeatureCollectionType" substitutionGroup="gml:AbstractGML"/> <complexType name="FeatureCollectionType"> <complexContent> <extension base="gml:AbstractFeatureType">

1173
s.jager
Influencer
smithgk
Contributor
smithgkContributor
asked in Data
CSV Reader parameters

Hi, I’m having some problems working with the CSV Reader. I created a Reader from a CSV that has 3 fields: Address, City, and ZIP. I have ‘Case Sensitive Field Names’ and ‘Enforce Strict Schema’ set to No, but then if I modify my source CSV to have the attributes: address, city zip, and Notes, the Reader fails to read in any data.What am I doing wrong here? It seems like even though I changes some properties for the Reader, it is defaulting back to the original schema that I input when creating the Reader. Any help is greatly appreciated. 

561
bwn
Evangelist
mr_fme
Enthusiast
mr_fmeEnthusiast
asked in Data
How to ignore errors in RCaller

Hi,I have one script in R running in RCaller, but the RCaller show some messages like erros but no erros, my script run properly. I would like to ignore these messages and generate one feature for the next transformer. How can I do this?  Thank´s

735
hkingsbury
Celebrity
gogopotter90
Contributor
gogopotter90Contributor
asked in Data
How to select only the modified tables in sde versions ?

Hello , I'm looking for an efficient way to compare the versioned SDE with the original SDE. I'd like to perform this comparison before running the compression and before posting the jobs in the database.I believe it would be time-consuming to read and compare all tables from both the versioned and original SDE. Is there a method that would allow me to read only the specific tables that have been edited in the versioned SDE?Thanks in advance FME 2021

523
hkingsbury
Celebrity
jokerfme
Contributor
jokerfmeContributor
asked in Data
Problem :Insert new points through FME has no UT-Attribute Data

Hello,I have posted my question in esri community but I did not get answer yet .i hope ,if someone could fix my issue  .I am inserting points into an SDE version using FME, but after creating the point, I'm unable to edit the UT Attribute Data. My company uses UT-Editor (developed from ArcMap), and typically, when I create a new point in UT-Editor, a window automatically appears to fill fields in the UT Attribute Data.some fields are necessary to fill in like Adresse and … However, since I'm inserting the points through FME, when I use the UT Attribute Data symbol to select  the point to try  fill the fields in, the window does not open. How can I resolve this issue? Is there a specific method that would allow each new point to linkage to UT-Attribute Data by FME ? or How could open the UT Attribute Data window in UT-Editor for each new point ,come from FME ?Thanks in advance . FME 2021 

300
jokerfme
Contributor
jeroen_hack
Contributor
jeroen_hackContributor
asked in Data
Saving Excel with Template to Sharepoint Sync Error

Hello fellow FME enthusiasts!We've run into in issue with writing Excel files to our sharepoint. We have a workspace that output's an Excel file.We've set it up to use a template to make sure it has the same easily readable layout for our users. We store the Excel file on our sharepoint. Our sharepoint folders are available locally on the PC through One Drive. So we store them to a location like: C:\Users\[User Name]\[Sharepoint name]\[Folder name]The issue we have is that every time we do this with the Template setting active in the Excel writer we get a sync error saying there is one file in out local folder and another one online. It's easy enough to fix this manually (One Drive gives us this option). But it is a manual action. If we write to the same location without the template setting in the Excel writer all works fine.We are working with FME version 2024.0.3.0 (build 24220)Has anyone experienced this before? And found a way to fix this?

421
jobvdnoort
Supporter
hayley.lam
Contributor
hayley.lamContributor
asked in Data
Running a Sharepoint Excel in my workbench results in OSError: [WinError 10093] Either the application has not called WSAStartup, or WSAStartup failed

Hi there,  I have connected to a Sharepoint Excel file in my workbench. I am able to read the rows and values when inspecting the Excel.  However, when I connect the Sharepoint Excel reader into any transformer, I get the error below: Exception ignored in: <finalize object at 0x13bf76a0; dead>Traceback (most recent call last):  File "weakref.py", line 590, in __call__  File "C:\Program Files\FME\python\urllib3\connectionpool.py", line 1135, in _close_pool_connections    conn.close()  File "http\client.py", line 1020, in close  File "socket.py", line 505, in close  File "ssl.py", line 1311, in _real_close  File "socket.py", line 499, in _real_closeOSError: [WinError 10093] Either the application has not called WSAStartup, or WSAStartup failedException ignored in: <finalize object at 0x13bf7cc0; dead>Traceback (most recent call last):  File "weakref.py", line 590, in __call__  File "C:\Program Files\FME\python\urllib3\connectionpool.py", line 1135, in _close_pool_connections 

1612
hayley.lam
Contributor
petergoedbloed
Contributor
petergoedbloedContributor
asked in Data
Flow can't access template files

Hi All,I have been building workspaces that get data from an api and write that data to Excel, using Template files. FME Flow can’t access the location of the template file, so what I did was: Make them a published parameter and publish. Then once one Flow, I change the parameters to the Flow resources path where I uploaded the template file as well. I do this in the Workspace properties. However, this doesn’t work, when running the workspace, the log says it can’t open the Template File and still pointing to the old/local location of the template file.What would be the right workflow here?

373
redgeographics
Celebrity
aaronp_osni
Contributor
aaronp_osniContributor
asked in Data
Using lists efficiently

Hi folks,I’m hoping this is relatively straightforward and I’m making this more difficult than it seems. I’m collecting images in the field and trying to store the images as blobs in a database. There can be up to 6 images for each property. In the below example only 3 images were collected. I’ve got to the point in my workspace where I’ve joined the images (as blobs) to the survey data, but I’ve got 3 features instead of one. I have a unique survey id. How do merge the data so that I have one feature, with up to 6 images stored as blobs? I’m trying and failing to use lists in the most efficient way possible; via the Aggregator then List Element Filter, then somehow bring the data back together, but this is the point I’m stuck on. Any help is greatly appreciated. Many thanks,Aaron

392
aaronp_osni
Contributor
marta.podsiad
Supporter
marta.podsiadSupporter
asked in Data
Save polygon as gpx

Hi,I’m trying to save a polygon as a GPX track and am running into an issue. Here’s what I’ve done so far:Converted the polygon to a line using GeometryCoercer Reprojected to EPSG:4326 Created track_id, source, and name attributesThe translation completes successfully, but when I try to open it, I only see the attributes and no geometry. Any idea what I might be doing wrong? 2024-10-29 11_22_11-Window.png

2036
marta.podsiad
Supporter
jokerfme
Contributor
jokerfmeContributor
asked in Data
Writing Data into an Arcsde Geodatabase ?

Hello  , this is my first post , and I hope  to receive an answer I have in FME an ESRI license that allows me to read from and write to an Arcsde geodatabase . I'm interested in knowing if anyone has successfully edited specific feature classes , made modifications in a workspace (such as adding data , deleting data or moving points to new positions), and then writed the output back to the the same feature classes after editing  by ArcSDE geodatabase  writer.After editing and modifying the data, I understand that in ArcMap, data typically undergoes compression, tracking which data is added and which is deleted.Is it possible to write the output directly to these compression tables ?  Is this step  necessary to do by FME ?Second question :  about writing into Relationship classes I read an  article https://support.safe.com/hc/en-us/articles/25407490028813-Writing-Geodatabase-Relationship-Classesthat uses an XML workspace as a template for writing to relationship classes, but I didn’t f

1567
hkingsbury
Celebrity
jacklonsdale
Contributor
jacklonsdaleContributor
asked in Data
Excel Reader showing formula values as missing

Hi all,As part of a development of some routines, I have created a many tab excel file where i inserted Formulae using the AttributeManager and the Cell.Formula naming convention. Due to the nature of the VLOOKUPS i used (dynamically i suppose), i could not do this calculation in FME's Arithmetic Editor.Opening the excel sheet shows the values i do want, given by the formula in the cells. However, when i now come to read this sheet in my next routine, the values are all shown as <missing>. Has anyone encountered this before or have any ideas?Thanks all, kind regards,Jack

14913
braggken
Supporter
timboberoosky
Contributor
timboberooskyContributor
asked in Data
How do I handle possible errors when writing multiple feature types with post data processing

I’ve alluded to my question on a few other posts about FeatureWriters:When FeatureWriter fails there is no output | CommunityAdd a Rejected port to FeatureWriter | CommunityBut figured I’d ask a more directed question in its own posting. I’m in a situation where I want to do some post-processing after writing data to each of my 40 feature types. I have a FeatureWriter for each feature type, in an attempt to catch and do no post-processing for feature types that fail, but continue with post-processing on feature types that write successfully. I’ve come to realize that any FeatureWriter that fails, stops the entire workspace. Setting “Rejected Feature Handling” to “Continue Translation” doesn’t apply to a failed FeatureWriter. The “Rejected port” link above is suggesting adding such a port to the FeatureWriter, which would solve my predicament, at least as I understand it, but it’s not implemented as of FME 2024.1.1, and I don’t know if there are plans to do so. I’ve also tested out sett

794
timboberoosky
Contributor
robert_punt
Contributor
robert_puntContributor
asked in Data
create time segments

Hello, i have a question for a scanner. the scanner drive through our city. the scannner makes a couple of rounds per day. But i have many record with a timestamp. between the rounds there is a time diffrent say 1 hour or more. how can i crerate a round number that the scanner makes.for example the scanner start at 7.00 en ends at 10.00  (thats round 1) and then starts at 13.00 and ends at 16.00. (thats round 2) the next day is somewhat the same but the count must go on. Beste Regards Robert

617
nielsgerrits
VIP
rhonda.edwards
Contributor
rhonda.edwardsContributor
asked in Data
LAS Writer: Failed to obtain point cloud data from feature

LAS writer: Failed to obtain point cloud from feature.  Only features with point cloud geometry are expectedLAS writer: An error has occurred. Check the logfile above for details Help. I’m receiving the two lines of error shown above ^ I am trying to modify a folder of .LAS data to create a TIN based on a certain classification (Ground). I don’t understand why I’m receiving these error codes if all of the other items are running and previewing successfully. FORM Workspace: Read in .LAS, Write out .LAS 

292
rhonda.edwards
Contributor
marta.podsiad
Supporter
marta.podsiadSupporter
asked in Data
xml does not open

Hi, I have downloaded data from WFS (GML) as xml (attached). It shows the location of a parcel. However, I am struggling to read information in FME and create a polygon representing parcel.I tried to read it as:OGC GML (Geography Markup Language)OGC GML SF-0 (Geography Markup Language Simple Features Level SF-0 Profile) I am getting this error XML Parser error: 'Error at file: 'C:\Users\MPODS\AppData\Local\Temp\FME_XSD_CACHE\fmexsd_1729842858321_34548.xml' line: 1 column: 1 message: invalid byte '‰' at position 1 of a 1-byte sequence'An error occurred while attempting to parse the XML Schema document 'C:\Users\MPODS\AppData\Local\Temp\FME_XSD_CACHE\fmexsd_1729842858321_34548.xml'The XML Module halted on error, see the logfile for detailsAn error has occurred. Check the logfile above for detailsFailed to obtain any schemas from reader 'GMLSF' from 1 datasets. This may be due to invalid datasets or format accessibility issues due to licensing, dependencies, or module loading. See logfile

9110
nielsgerrits
VIP
thomastupuivao
Contributor
thomastupuivaoContributor
asked in Data
using 1st row for column header for a variety of text files

I have a request to build a FME workspace that can read a text file and load it into an Oracle database with the filename with a suffix appended as the table name (ie. filename_ext).That is fine, the problem is the column headers and fields will differ, depending on the filetype (file A might have 10 columns, file B will have 17 and the column names will be different between A and B)I’ve considered using a tester that directs the flow to an attributemanager based on a “user parameter”. This becomes problematic if the source text file changes column structure or an additional file type (file C, D and E) is introduced.Is there a way for the text file to be read (as generic in the featurereader), have the attributes exposed, split and output with the 1st row as the column headers. Ideally without having to know the column names prior.Thanks Thomas

1482
thomastupuivao
Contributor
dor1411993
Contributor
dor1411993Contributor
asked in Data
Error with reading Pipe and Structure Data Fields (Attributes)

Hello FME Community!I’m trying to read a Civil 3D file in FME, but I’m not able to access the specific attributes I need.I'm trying to retrieve Pipe and Structure Data Fields (Attributes), but I'm running into errors. Here are the errors that I’m seeing:```2024-10-27 15:13:47|   5.1|  0.0|ERROR |AutoCAD Reader: An error occurred for dataset 'C:\Users\Dor\OneDrive\Perach\Ayalon\חומר ראשני\R6407_01-RME-DR-SVD.dwg', in function AutoCAD2004Reader::readC3DSchema. Exception thrown: 'Not that kind of class' (Object of class AcDbProxyObject can't be cast to AeccDbNetworkPartDef.)2024-10-27 15:13:47|   5.1|  0.0|ERROR |An error has occurred. Check the logfile above for details2024-10-27 15:13:48|   5.6|  0.0|ERROR |Failed to obtain any schemas from reader 'CIVIL3D' from 1 dataset. This may be due to invalid datasets or format accessibility issues due to licensing, dependencies, or module loading. See logfile for more information.2024-10-27 15:13:48|      |     |ERROR |Workspace generation faile

340
dor1411993
Contributor
matthewb
Contributor
matthewbContributor
asked in Data
Conditional Adjacent Feature Value Grouped by Attributes

I have a dataset that has customer data, vehicle data, planned driving distance (with a bunch of other scrubbed attributes).  In this data, there are many days when a customer was not planned to have accumulated any distance but did use a machine.  If the planned driving distance to date is missing, I want to copy the value for the previous planned_Driving_dist_to_date that has a value, but I want this sorted by date and grouped by customer and machine.   The attached spreadsheet shows my dataset. 

281
matthewb
Contributor
mr_fme
Enthusiast
mr_fmeEnthusiast
asked in Data
"Shadown" in final Mosaic

Hi,I'm using the Raster Mosaicker to create a mosaic with overlapping images. However, the final result generates 'shadows' as shown in the image below. What is the best configuration or additional transformer I can use to solve this problem?Thank you 

394
mr_fme
Enthusiast
E
erinjayaustin
asked in Data
GEOTIFF Writer Interleaving Type option doesn't seem to work

When writing to Geotiff I would like Band interleaved by Pixel (BIP, interleaved RGBRGB) rather than Band Sequential (BSQ, per channel RRGGBB) interleaving but whichever option I use, the Geotiff always saves using Band Sequential interleaving.I have checked using Photoshop and also GDALinfo and both agree that INTERLEAVE=BAND rather than PIXEL. Is this a bug/can anyone else replicate this?

1553
oscard
Influencer
nsmart
Participant
nsmartParticipant
asked in Data
Reading CSV and Writing to MSSQL Database: tables not recognized by geometry_columns

Hello :)Here is what I am doing: I am working with FME-Flow and FME Workbench 2023.1.1. I have geodata in the form of .csv-files and am using a simple CSV-Reader ➡ MSSQL-Writer to create my datatable in my MSSQL-database (see attached image for FME-script). The datasets are saved to the ‘dbo’-schema and work without problem in GIS (QGIS in this case). This is done for a number of CSV-files. As an example I have shown the summary annotation for the dataset ‘ov_boende_aldre’.What’s the issue? When FME writes to the MSSQL database, the resulting datatable(s) is/are not recognized by the systems view ‘geometry_columns’ (se image below). Why not? When creating a dataset directly in MSSQL or in GIS (QGIS, for example), the dataset appears in said systems view. What can be the issue here? Any secret sauce to make FME create the dataset correctly that I am just missing?All help and hints are appreciated. Also, if you are aware that there is a similar post with a solution then do please direct

432
bwn
Evangelist
i_miss_the_chat
Contributor
i_miss_the_chatContributor
asked in Data
loop through the newly created buffers and associate them with subsequent points that fall within the new buffer.

Hello,I am creating buffer polygons for points, but I would like to loop through the newly created buffers and associate them with subsequent points that fall within the new buffer.If 2 points are next to each other, the first point would be buffered and the second point would be associated with that buffer.Thank you.

391
bwn
Evangelist
gogopotter90
Contributor
gogopotter90Contributor
asked in Data
How to create a Block diagram of Geodata

Hello ,How could i convert Geospatial Electrical Network Data (includes stations, cables , houses) are connecting to each other to convert these data as compact Block diagram .it is not important the coordinates ,I need to create a simple block diagram  of electrical data .I would like to adjust the positions of elements by bringing them closer together without overlapping and create the suitable connections between them .Is there any idea that I should to test ?FME 2021thanks in advance .

633
gogopotter90
Contributor

Badge Winners

  • Welcome Back
    daposadikuhas earned the badge Welcome Back
  • FME Flow Certified Professional
    lubohas earned the badge FME Flow Certified Professional
  • FME Certified Professional
    lubohas earned the badge FME Certified Professional
  • Student (Grey)
    kevinkieshas earned the badge Student (Grey)
  • Teacher (Grey)
    kevinkieshas earned the badge Teacher (Grey)
Show all badges

Community Stats

31,483
Posts
119,358
Replies
39,104
Members

Latest FME

Terms & ConditionsCookie settings
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.