Skip to main content
  • EN

Data

Using readers and writers to extract and load data.

  • 6,988 Posts
  • 25,736 Replies
6988 Posts
Jonathan Slope
Contributor
Jonathan SlopeContributor
asked in Data
ChangeDetector gives confusing results

Hi all, I've got some confusing to clear, and hope to find an answer here. I have 2, completely the same datasets, uploaded to ArcGIS Online. Now, i want to update them. To test it, i took 2 completely identical layers, read them from AGOL and now, it shows differences. Can someone explain me what the problem is? If i only check on attributes, then there is nothing wrong, but if i also check on geometry, there is somewhere a change, and the Changedetector gives updates as output, but it should be unchanged. All the data is 2D, there is no 3D. In the update detection key attributes, i selected all the attributes, but left GlobalID and OBJECTID out of it. same for check attributes. Even more confusing, if i do the same process in a new workbench, with the same sources etc, i get different results. Sometimes it igeve met 900 updated, and somethimes 1322Thanks in advance!

15911
Jonathan Slope
Contributor
rudy_v
Contributor
rudy_vContributor
asked in Data
Error running translation - reading SQL Server Spatial data on Flow and Form - 2024.1

Hi I am doing migration testing from 2023.1.2 and 2024.0.3. It seems to completely fails on 2024.1.I have tested it on 2023.1.2 and 2024.0.3, worked perfect - no issues.It fails on FME Flow and FME Form on the server - but works OK on my local PCon the server - oldeb 2019, clr 2019, are installed, as with .net 4.7, 4.8 andIt is just a simple sql server spatial reader - and a bit of reprojection - it fails on reading the reading the dataCheersRudy

1788
anthrax_79
Contributor
yahyanar
Contributor
yahyanarContributor
asked in Data
Open an ALKIS data

I am trying to read an ALKIS data file coming as XML file. I have two files, the first one come with version 6.0 and FME opens this file without any problem (xmlns="http://www.adv-online.de/namespaces/adv/gid/6.0"). However, when i try to read the version 7.1 (xmlns="http://www.adv-online.de/namespaces/adv/gid/7.1") I got an error as following:XML Parser error: 'Error at file: 'K:\……..line: 1 column: 1 message: invalid document structure'An error occurred while attempting to parse the XML Schema document 'file:///K:/Geodaten/Projekte/ALKIS/2411_ALKIS_NAS/NAS-Operationen.xsd'The XML Module halted on error, see the logfile for detailsA fatal error has occurred. Check the logfile above for detailsFailed to specify the feature index as constraints on the readerFailed to obtain any schemas from reader 'NAS' from 1 datasets. This may be due to invalid datasets or format accessibility issues due to licensing, dependencies, or module loading. See logfile for more information do you have any so

602
yahyanar
Contributor
oliver.morris
Contributor
oliver.morrisContributor
asked in Data
DuckDB Summarize command

I am trying to use the duckdb summarize commandhttps://duckdb.org/docs/archive/0.9/guides/meta/summarizein an in memory duckdb using the sqlexecutor:SUMMARIZE SELECT * FROM read_parquet('/Users/x/Downloads/Address.parquet');This doesn’t return any results/attributes but ‘runs’ successfully in FME. In DBeaver it runs no issues.Any idea is there is a trick to get this working?Many ThanksQ!

854
oliver.morris
Contributor
lambertus
Enthusiast
lambertusEnthusiast
asked in Data
PostGIS Reader - warning about 3D feactures

I created a 2D spatial point dataset in a PostGIS database.Today I tried to load them into FME, that works perfect. I get the right locations including the according attributes.I only get this warning:Dimension metadata is ambiguous for spatial column 'geom'.  Reporting 3D when 2D+measures might be presentDid I forget to set some specific settings while creating this dataset? There should be no 3D data present, I only saved 2D data :)

673
crystalatsafe
Safer
lifalin2016
Contributor
lifalin2016Contributor
asked in Data
What versions of OGC Geopackage and SQLite does FME produce ?

I'm having a hard time determining which version of OGCGEOPACKAGE that FME produces ? It's not in the writer doc, so I'm asking here.And subsequently, what version of SQLite is produced? (newest is 3.39.4 from Sep 29 this year)As far as I can gather, the versions has been fixed since 2020.0? Is this true?In the "gpkg_extensions" table I found a reference to "spec120", which may (or may not) indicate, that the Geopackage version is 1.2.0, a fairly old version (newest is 1.3.1).And the real question is what extensions are the SQLite database born with? Clearly not "foreign _keys".Can one update some template database if one needs a newer version/more extensions?Cheers.

442
gedgg
Contributor
nicholas
Contributor
nicholasContributor
asked in Data
How to SELECT all records with Point geometry?

Hello all,I have an Oracle Spatial table that contains some records with a Point geometry and some records with a Polygon geometry.Considering the large number of records, I would like to take advantage of the ability of the Oracle Spatial Reader to perform a SELECT within the Oracle database and return only records with a Point geometry(This is preferable to Reading all records and then using a GeometryFilter for performance reasons)How to I write the SELECT statement?I assume it has something to do with SDO_GEOMETRY, but I just cannot get the syntax correct

263
geomancer
Evangelist
ldreyfuss
Contributor
ldreyfussContributor
asked in Data
Portal Service published from FME Form doesn't have data in the fields.

I’m using the ArcGIS Portal writer to publish point data to a portal. The dataset publishes successfully in FME and all of the points appear on the map, but the attributes don’t show up in the table. Removing specific fields doesn’t seem to help. Has anyone seen anything like this before? 

333
ldreyfuss
Contributor
zzupljanin
Contributor
zzupljaninContributor
asked in Data
Insert external reference

Hi,I've been trying too use autocad_xref_path and autocad_xref_name to insert file (another dwg) into REALDWG writer feature type. It should be basic. But it's not working. No warnings, no errors.Did anyone succesfully managed to do that? Can you explain me what am I doing wrongRegards,Želimir

7916
saraatsafe
Safer
fme_hans
Contributor
fme_hansContributor
asked in Data
Dynamic reading and writing tables

Hi,I would like to create a workspace that reads all tables that start with the same prefix. I then want to save the result per table as a shape file. If a table is modified by a add or remove an attribute, the workspace should pick this up automatically. Is that possible? My first thought was to work with a dynamic reader, but then I get stuck that it will retrieve all tables, including in other schemas within the database. The second approach is, I create a query that generates select statements and puts these statements in a text file. I then read the text file with a textline reader and install an SQLExecuter. That's all going well.What I run into is that the SQLExceuter cannot automatically expose all attributes.

13710
fme_hans
Contributor
K
kd
asked in Data
PostgreSQL reader: bad_expected_access

Hello everyone,I encountered a similar issue as the one described in the following post:https://knowledge.safe.com/questions/95574/problem-with-postgis-reader-in-fme-2019-infinity-v.html Basically I had an invalid date in my PostgreSQL table (no idea how it got there). It had a value of 201910-11-10 which of course is not valid whatsoever. FME 2020 ( Build 20238 - WIN64) throws an error when you try to read this table using a Reader, a FeatureReader or an SQLExecutor/creator.I was able to bypass this by adding a where clause upon reading (e.g. mydatefield < '2021-01-01').I'm not looking for a solution for my problem here. I just wanted to add this to the forum so that people who encounter a similar issue know how to solve it. The FME software currently throws an error which is not really useful for debugging purposes (POSTGRES reader::bad_expected_access).

1652
stezi
Contributor
mccunee
Contributor
mccuneeContributor
asked in Data
Parquet File not Readable over s3 because of slashes

I have many parquet files stored in an s3 bucket; when I use the s3 connector to download the parquet file, I am able to read it successfully, but passing the s3 uri (“s3://”) raises an error:PARQUET reader: PARQUET reader: Failed to open file 's3:\my-bucket-s3-test\fme-test\test.parquet' for reading.  Please ensure that the file exists and you have sufficient privileges to read itFailed to obtain any schemas from reader 'PARQUET' from 1 datasets. This may be due to invalid datasets or format accessibility issues due to licensing, dependencies, or module loading. See logfile for more informationWhen I try to pass the url:PARQUET reader: PARQUET reader: Failed to open file 'HTTPS:\s3.us-west-2.amazonaws.com\my-bucket-s3-test\fme-test\test.parquet' for reading.  Please ensure that the file exists and you have sufficient privileges to read it When I use the feature reader, regardless of whether or not I specify a web connection, it converts forward slashes to back slashes. How do I work w

813
hkingsbury
Celebrity
anthrax_79
Contributor
anthrax_79Contributor
asked in Data
FME Form 2024 and Microsoft SQL Server Issue

Hi, I have downloaded, installed and uninstalled all versions of FME Form from 2024.0 up to latest 2024.2 and still having issues reading and or creating a new reader from Microsoft SQL Server. Every time I create a new and or update reader transformer, the reader disappears. Same when I create a new reader, it never show up. I have another machine that runs (2023.1.1.1 (20231011 - Build 23636 - WIN64) and it runs fine. I want to update my version as this old version is unable to read from Amazon RDS. Has anyone experienced this or experiencing this?Update * after further investigations. I have found the below. There seems to be incompatibility with the versions and I could not find a version that can read both version of MSQL. Microsoft SQL Versions FME Form 2023 (x64) FME Form 2024 (x64) Microsoft SQL Server 2017 - 14.0.3480.1 (x64) TRUE FALSE Microsoft SQL Server 2019 - 15.0.4410.1 (x64) FALSE TRUE

560
anthrax_79
Contributor
jorge_rosales
Contributor
jorge_rosalesContributor
asked in Data
How to know the feature class and annotation class linked by a relationship class in an ESRI FGDB?

Is it possible to know that the relationship class "C04_EjesVia_LS_Rel" links the feature class "C04_EjesVia_LS" with the annotation class "A04_EjesVia_LS"? Any specific Reader configuration or attribute that I'm missunderstood?I attach FGDB sample. 

473
crystalatsafe
Safer
emerson_styles
Contributor
emerson_stylesContributor
asked in Data
How to split attribute into two lists

I have a Geometry column that includes an easting and northing that is separated by a space in the middle as follows: I want to split this data up so that eastings go into one column and northings go into another. I have tried using an AttributeSplitter with a space as the delimiter and then a ListExploder to view the results but unfortunately this doubles the number of records in my data instead of creating two unique columns. How can I create two separate columns from my data?

1076
geomancer
Evangelist
georedyo
Contributor
georedyoContributor
asked in Data
How to instaure a default value to continue the translation

Hi, i’m working on water treatment facilities. I’ve to categorize them in order to write a excel file which lists the existing and some deliverable.  I’ll try to explain the best i can : basically the data are from a gdb,  i’ve a tester to sort depending on categories. Some cat can have missing/null values, i need to “force” those missing values into the translation in ordrer to write the corresponding categories in the excel files.my situation now is that the missing values given they’re not written shift the writting in my writer and then do not correspond with their according row.i’ve 9 differents categories so 9 row but when i’ve just 5 filled categories, i’ve those 5 written but it change the alphabetical order in the written row so the categories dosent match the row. How can i say to the translator “if i’ve no values write the default value “0” else “no action”” Thanks, hope i’ve been clear enough

707
geomancer
Evangelist
fmelizard
Safer
fmelizardSafer
asked in Data
Maximo Connector - what do you want it do?

Hi All,We are currently working on Maximo connector transformer(s), and have been successful in connecting and querying a Maximo database. However, there is an incredible amount of functionality available through the APIs, and I would like to initially concentrate on what is most useful to our users.So I am requesting your feedback on how you would like to use FME with Maximo. Do you want to add Assets or Work Orders? Do you want link Maximo info to your GIS or BIM data. Do you want to do complex queries? This is a chance for you to shape the design of a new series of transformers.Detailed uses cases would be greatly appreciated.

258
anthrax_79
Contributor
tbarker1986
Contributor
tbarker1986Contributor
asked in Data
Issue with Dynamic Writing to SDE for larger datasets

Hi all, I am experiencing an issue with writing dynamically to SDE for large datasets (i’m unsure of the exact number, but above ~5000 features seems to cause a fail).  My flow reads in a dataset name from an SQL table, matches it to a dataset that has been downloaded as part of a previous flow, checks if it has been updated and if so drops and recreates the Feature Class using a name given in the SQL tables.When running the flow in batch using a Workspace Runner, ~half of my datasets ran successfully, and ~half failed to be updated in the SDE. I have looked into this is some depth and the only pattern I can see is the datasets that are failing are larger. The error message I am getting is below. I have tried using a sampler on the data just before the FeatureWriter and the flow works correctly, which validates my findings. What could be the potential reasoning behind this behaviour? or has anyone else experienced similar? If there are any other suggestions of settings I could change t

453
nielsgerrits
VIP
D
dkirouac
asked in Data
ArcGIS Online re-authentication

We have several FME Server services that runs nightly to update hosted services. This all works well but every month or two the services all fail because they have lost their authentication to AGOL even though there were no account changes. I can re-authenticate the connection and then the services run for a month or two and then they lose their authentication again. Is there a way to set up the authentication so that they do not have to be periodically re-authenticated? Thanks in advance,David

43117
gistech_
Contributor
arnold_bijlsma
Enthusiast
arnold_bijlsmaEnthusiast
asked in Data
What are fast Geospatial Formats to Read?

A bit of a subjective question, but what are fast file-based geospatial data formats to read in FME?I have a set of geodatabases (10GB) covering the whole country, and it’s located locally on the C-drive of  the FME Flow server. Users specify a bounding box to extract the data for their area of interest.To make the current setup faster, I have played with indexes and made sure that I’m doing a bounding box intersect on the FeatureReader, etc. But it still takes at least 10-15 minutes to get the output.One other idea that I’m playing with is to use a different format for the source data instead of geodatabase. GeoPackage got a mention somewhere; FlatGeoBuf apparently is fast; and I have heard good stories about GeoParquet. Any other formats I should consider?Has anyone done any comparison on this? Any pitfalls / drawbacks I should be aware of?At the moment I’m looking more at file-based formats, as formats that require additional (database) software would involve IT too much.I will do t

38614
tva
Contributor
jokerfme
Contributor
jokerfmeContributor
asked in Data
issue - Arcsde Writer (Row Match)

Hello ,I need a help I successfully updated a point geometry in the ArcSDE writer within a specific version. I encountered an issue.When I select "Update Mode" and set "Row selection - Columns " to ObjectID, I get an error translation  running  . But if I leave "Row seletion - Columns" empty, the translation runs successfully. The "Row selection - Columns " field turns red when left empty, and I’m unable to click "OK." To work around this, I temporarily set "Row Selection - Columns" to ObjectID, click "OK," and then use a Remover transformer before the writer to remove ObjectID—and when I run the workspace ,it  works great and translation is successfully . But ’m unclear on how FME identifies the correct point to modify it ,how to resolve my issue ? Update Spatial columns : Should be yes or no ?FME version: 2021, 64-bit.Thanks in advance

1087
bwn
Evangelist
lambertus
Enthusiast
lambertusEnthusiast
asked in Data
UserParameter values as attribute name

I would like to use 2 UserParameter values as attribute names in my workbench. Is this possibe?Current situation:What I would like to have: 

481
veschwab
Contributor
spalrobert
Contributor
spalrobertContributor
asked in Data
QGIS python library not able to load

Hi, I am trying to run a custom python code that uses QGIS library in PythonCreator. However when I run the code I get this error:Python Exception <ImportError>: DLL load failed while importing _core: The specified procedure could not be found.I tried everything, I followed the documentation/manual how to setup custom python interpreter, I tried to setup dependencies and environment variables but none of that worked. Can someone tell me what am I doing wrong? Thanks!

652
spalrobert
Contributor
marjan_nikoukar
Contributor
marjan_nikoukarContributor
asked in Data
Daily Data update in FME

Hi everyone,I’m working on an FME workbench where I download a full dataset file geodatabase daily. I also have a CSV file that lists daily changes.My goal is to remove files from the main dataset based on the CSV and add updated files from file geodatabase back into the main dataset.I’ve tried using the FeatureMerger and FeatureJoiner transformers to identify changes, but they were not effective due to time constraints. Since this needs to be done daily, time efficiency is crucial.Do you have any suggestions, Python scripts, or SQL queries, or any other ideas to speed up this process?Thank you!

728
redgeographics
Celebrity
tunz
tunz
asked in Data
write Paramids when converting to mbtiles

Hi, Is there a way to remove build pyramids when converting geotiff images to mbtile? I just need the image to display and improve mbtiles creation time 

181
hkingsbury
Celebrity

Badge Winners

  • Student (Grey)
    lukeh81has earned the badge Student (Grey)
  • Student (Grey)
    cbateshas earned the badge Student (Grey)
  • FME Certified Trainer
    erik_janhas earned the badge FME Certified Trainer
  • Integrate Spatial Data
    laisgouldhas earned the badge Integrate Spatial Data
  • Student (Grey)
    wthallhas earned the badge Student (Grey)
Show all badges

Community Stats

31,241
Posts
118,445
Replies
38,833
Members

Latest FME

Terms & ConditionsCookie settings
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.