- - Community Resources
Ideas
Product Updates
Events
Blog

Home
Forums
FME Form
Data

Data

Using readers and writers to extract and load data.

6,782 Posts
24,942 Replies

When you subscribe we will email you when there is a new topic in this category

6782 Posts

Recently active

Most replies Most views Newest first

arnold_bijlsmaEnthusiast

asked in Data

What are fast Geospatial Formats to Read?

A bit of a subjective question, but what are fast file-based geospatial data formats to read in FME?I have a set of geodatabases (10GB) covering the whole country, and it’s located locally on the C-drive of the FME Flow server. Users specify a bounding box to extract the data for their area of interest.To make the current setup faster, I have played with indexes and made sure that I’m doing a bounding box intersect on the FeatureReader, etc. But it still takes at least 10-15 minutes to get the output.One other idea that I’m playing with is to use a different format for the source data instead of geodatabase. GeoPackage got a mention somewhere; FlatGeoBuf apparently is fast; and I have heard good stories about GeoParquet. Any other formats I should consider?Has anyone done any comparison on this? Any pitfalls / drawbacks I should be aware of?At the moment I’m looking more at file-based formats, as formats that require additional (database) software would involve IT too much.I will do t

10911

4 hours ago

marjan_nikoukarContributor

asked in Data

Daily Data update in FME

Hi everyone,I’m working on an FME workbench where I download a full dataset file geodatabase daily. I also have a CSV file that lists daily changes.My goal is to remove files from the main dataset based on the CSV and add updated files from file geodatabase back into the main dataset.I’ve tried using the FeatureMerger and FeatureJoiner transformers to identify changes, but they were not effective due to time constraints. Since this needs to be done daily, time efficiency is crucial.Do you have any suggestions, Python scripts, or SQL queries, or any other ideas to speed up this process?Thank you!

203

4 hours ago

fme_hansContributor

asked in Data

Divide planes by area sum

Hi,I have a layer with closed polygons. On top of these polygons lies a layer with smaller areas. These areas are divided into 3 types.Each type of area needs to be assigned a year, and the total area per table type should be approximately equal within the larger area.How can I create a distribution where each type of area is used, and the total area of each type is approximately equal?How could I achieve this with FME?

472

17 hours ago

marjan_nikoukarContributor

asked in Data

Convert annotation class to polyline or point in FME

Hi,I have a annotation class in a file geodatabase. I want to convert it to a polyline or point in a file geodatabase. I could not fine any transformers in FME. I really appreciate your assistance.

how to read ssf data format in fme

748

1 day ago

spalrobertObserver

asked in Data

QGIS python library not able to load

Hi, I am trying to run a custom python code that uses QGIS library in PythonCreator. However when I run the code I get this error:Python Exception <ImportError>: DLL load failed while importing _core: The specified procedure could not be found.I tried everything, I followed the documentation/manual how to setup custom python interpreter, I tried to setup dependencies and environment variables but none of that worked. Can someone tell me what am I doing wrong? Thanks!

Optimizing i3s (slpk) LOD

I'm converting 3D data from Postgres to an Esri Indexed 3D Scene Layer (slpk). I'm converting this to an i3s service on Azure (not ArcGIS Online). When loading this as an external layer in ArcGIS Online, the LOD seems totally random, buildings are being visualised but not the way I'd like to see (not based on distance). Is there a way of optimizing this in FME?

Writing ArcGIS Geodatabase attachments - origin row error

I am working with a script that transfers data from one SDE to another SDE. Currently, the script does not ensure that the relationship between a feature and its attachment remains intact. To address this, I extended the workspace based on the guidelines in this article: Writing ArcGIS Geodatabase Attachments.This solution worked fine when transferring data from one GDB to another GDB using the same data model. However, when we run the script from one SDE to another SDE with the same data model and data, it fails and I encounter the following error messages: Failed to retrieve origin row for a relationship feature with OBJECTID '3' from table/feature class 'BAGGERWERK_WATERLOPEN' GeodatabaseRelationshipFeaturesPipeline::GeodatabaseRelationshipFeatures (SortFactory): A fatal error has occurred. Could you provide guidance on how to resolve this issue?

brunoContributor

Hello,I would like to know if the IfcZone are managed with the version 2024?Best regardsBruno

New Snowflake geometry data type (not geography)

Snowflake recently released their new "geometry" data type. Not to be confused with the existing "geography" data type. The geometry data type can contain geometries in other coordinate systems, not just WGS84. Whereas geography is specifically WGS84. See Geography vs Geometry data types documentation hereGeospatial Data Types — Snowflake Documentation When will FME support writing of the new snowflake geometry data types? Is there a way to do this right now, or will it be a future release of the Snowflake Spatial writer?

628

2 days ago

sharvikhawteContributor

asked in Data

3D voxel for geological model

I have 4/5 layers of rasters representing the geological horizons. I want to visualize the layers in 3D voxel. what the result look something like this. Is there a way to get this with fme?

262

3 days ago

anne_kathr1neContributor

asked in Data

Writing features to ArcGIS Online

Hi, Im trying to write approx 17.000 rows into an existing feature service in AGOL. I get the error "entity too large for url", so I assume its too many rows. I have tested with a few rows, and that works fine. I have tried now to use the ModuloCounter and a TestFilter to split into smaller groups, and then add multiple AGOL-writers, so they only write one group at the time. Im not sure this is the best way to go, as I would the need a lot of writers. Another is to use the FeatureWriter to do the same, but I would still need to do the setup for all the groups, so not sure if thats ideal. Does anyone have any better suggestion? Best,Anne Kathrine

How to set extents on DWG ouput file

I am writing AutoCAD DWG with changing areas/extents. My template file defines the styling. When opening the file in CAD it zooms to a blank area (possibly from the extents of the template file) and I need to zoom to extents to find my current area. Is it possible to either remove any extent information from the template file, or force the output DWG file to zoom in on the current area extents?

544

5 days ago

rudy_vContributor

asked in Data

Error running translation - reading SQL Server Spatial data on Flow and Form - 2024.1

Hi I am doing migration testing from 2023.1.2 and 2024.0.3. It seems to completely fails on 2024.1.I have tested it on 2023.1.2 and 2024.0.3, worked perfect - no issues.It fails on FME Flow and FME Form on the server - but works OK on my local PCon the server - oldeb 2019, clr 2019, are installed, as with .net 4.7, 4.8 andIt is just a simple sql server spatial reader - and a bit of reprojection - it fails on reading the reading the dataCheersRudy

1296

6 days ago

zachwrightContributor

asked in Data

Writing Data to SQL Spatial Database Results in Successful Translation With No Actual Data Written/Updated

I have been having a consistent problem in FME for quite a while now, across both 2023 and current 2024 versions. I have a workspace set up which has a SQL Spatial reader, a few transformers, then a writer back to the database of origin.Any time I run this database, it goes through and successfully runs it within FME. However, as soon as I go to another source and read the database table(s) involved, such as SQL Server Management Studio or QGIS, I find that the data is still the exact same as it was originally, like FME had done absolutely nothing. Likewise, if I make a new table, it will simply make an empty table with only the columns/fields intact, with no attributes at all. I have no problems editing data from this database in QGIS or SQL Server Management Studio and have even had colleagues run the exact same FME Workspace with no such problems. I have also had two different FME installations on two different devices behave exactly like this.I'm pretty much out of ideas here. Has

294

7 days ago

cnorthContributor

asked in Data

Insert Salesforce issue with related record (Sharing record)

Hi, Currently have a process that recieves a list of record id’s from Salesforce using HTTP callouts, the record ID include the ID of records from a custom object and the id of a number of users, we are trying to create an insert in the customobject_share to create records to give the list of userId access to the records from out custom object. We currently are getting an error,Salesforce Writer: Record 1 was not processed successfully: 'FIELD_INTEGRITY_EXCEPTION:invalid parent id: Parent ID:ParentId --' The parent id is the record Id of the salesforce record we are creating a related share record against. the record exist in salesforce and in testing we have event tried hardcoding te ID of both records, and user Id with RowCause = “Manual” and Access Level = “Edit”. The form is running as using the Salesforce reader/writer with a System Admins login details? Thanks

121

7 days ago

stfcjoeParticipant

asked in Data

ArcGIS Online Feature Service Writer: Error 405

Hi there, I keep getting this error message: ArcGIS Online Feature Service Writer: Encountered an unexpected error. The error code from the server was '405' and the message was: 'This SqlTransaction has completed; it is no longer usable.'. Details: 'Unable to apply edits. Please check your parameters.' Any ideas? It’s attempting a truncate. I have purged temporary files and tried to replace all the connections and make a copy of the workbench but still no dice. Thanks,Joe

782

8 days ago

sharvikhawteContributor

asked in Data

Uploading raster (tif) in Arcgis online using FME

I am trying to upload a tif stored in my PC locally to the contents in arcgis online. I can upload the shapefiles efficiently but not tifs. Can someone advise? Thanks!

494

8 days ago

pmcphersonContributor

asked in Data

Warning reading Snowflake Spatial Data

When using a FeatureReader transformer to read Snowflake spatial data, the following warning is posted into the log.Snowflake Spatial Reader: Skipping data in column '.GEOG' due to error (JDBC Type = VARCHAR, Vendor Type Name = GEOGRAPHY, Error: class java.lang.String cannot be cast to class [B (java.lang.String and [B are in module java.base of loader 'bootstrap'))The process does complete normally, but the geometry is not read automatically. Has anyone else encountered a problem like this? I’m using FME Form 2024.0.1.0, Build 24202.

150

8 days ago

nanchuangyeyuContributor

asked in Data

How to create a Esri Personal Geodatabase whose structure is specified in an Excel table? ?

Hi everyone, I want to create a Esri Personal Geodatabase whose structure is specified in an Excel table. Is that possible? How can I achieve that?

604

8 days ago

carmijoContributor

asked in Data

Reading ArcGIS Online Feature Service shared with the organization - not owned by the user or shared with a group.

Using an Esri ArcGIS Online connection and the Esri ArcGIS Online (AGOL) Feature Service reader, I'm unable to read a feature service that has been shared to our organization.When looking at the select form, i’m only provided my user folder and a groups folder. The feature service i’d like to read isn’t owned by me or shared with a group. Is this as-designed or is there another way to read it? Thank you

452

10 days ago

marjan_nikoukarContributor

asked in Data

How to update a dataset using a change table in FME

Hi everyone,I have a feature class in a File Geodatabase that I need to update. I have two sources for these updates: a CSV file with codes of the changed features, which I should use to remove those features from the main dataset, and separate shapefiles for each changed feature that needs to be added back to the main dataset. (I shouldn’t delete and update the whole dataset, I should update the changed features only.)What is the best way to approach this in FME?Thank u in advance for your time.

735

12 days ago

tonoContributor

asked in Data

Destination Feature Type Routing Correlator

Hi,When running a workspace with cache mode activated i can see in the Translation log that fme form writes ffs, and writes and index and so starts it with a process i am not familiar with: “Destination Feature Type Routing Correlator” and it takes so long to go through.What is this process about? is it possible to avoid?Thanks in advance!

201

13 days ago

gboquinContributor

asked in Data

SQL Server failing in SQL Executor in FME Form

Hi, I have a SQL Executor calling a stored procedure. I am running FME Form FME(R) 2023.2.1.0 (20240104 - Build 23774 - WIN64) and am getting this issue:Microsoft SQL Server Non-Spatial Reader: Failed to connect using Microsoft OLE DB Driver 19 for SQL Server. Provider error '(-2146824582) Provider cannot be found. It may not be properly installed.'The transformer is not failing, but its not producing a result either. Not sure how to move forward with this one. Any thoughts?Thanks

382

14 days ago

marjan_nikoukarContributor

asked in Data

Split attributes and write them as feature classes in a new file geodatabase

Hi, I used an AttributeFilter transformer to split attributes. I want to write each of them as a feature class in a new file geodatabase. What is the best way to do that?Thank you for your assistance.

381

14 days ago

Page 1 / 272

Badge winners

vojtechmyskahas earned the badge Welcome Back
delaneytoursihas earned the badge Student (Grey)
gavinparkhas earned the badge FME Certified Trainer
TandraAtSafehas earned the badge GIS Day 2024
eqwansyafiqhas earned the badge Webinar Watcher (Grey)

Show all badges

30,141: Posts

Terms & Conditions Cookie settings

Sign up

Already have an account? Login

An FME Account is required to contribute

Login to the community

An FME Account is required to contribute

Enter your E-mail address. We'll send you an e-mail with instructions to reset your password.

Enter your e-mail address

Back to overview

Scanning file for viruses.

Sorry, we're still checking this file's contents to make sure it's safe to download. Please try again in a few minutes.

This file cannot be downloaded

Sorry, our virus scanner detected that this file isn't safe to download.

A product of

Company Products Giving Back Careers Contact Privacy Feedback

Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.