Skip to main content
  • EN

Data

Using readers and writers to extract and load data.

  • 7,083 Posts
  • 26,087 Replies
7083 Posts
O
ocarey
asked in Data
setting up user parameters for readers

I have created a workspace which has a prompt for user parameters, as I want the user to be able to run the workflow on different sets of data. The workspace reads in a set of CSVs which have identical schema, performs some transformations and then writes the data out into a single CSV. The process works for what I want it to do. The problem is when the prompt comes up to run and I choose a different set of CSVs, these CSVs have the same schema as each other but not the same schema as the previous set of data. The attributes that are used in the transformations are the same but other attributes are different and this data does not get read in. I want the reader to read in the new schema each time, but it always retains the schema from the first set of data I used to set up the reader. How can I fix this?

4190
thijsknapen
Contributor
aegman
Contributor
aegmanContributor
asked in Data
Dynamic connection file in FeatureWriter ESRI Geodatabase (ArcSDE Geodb)

Hi, I've configured a FeatureWriter to dynamically write feature classes to an ESRI Geodatabase, but I need to be able to write different feature types to different databases. I've configured the FeatureWriter to get the connection file from an attribute, but it only uses the first connection file attribute it receives, it doesn't change connection based on the attribute value of the incoming features. Is there a way to get it to do this, or do I need to setup a writer for each possible database connection?

270
aegman
Contributor
W
whoffman
asked in Data
ArcGIS File GDB Writer Slow

I made a simple workflow to compare two datasets using the Change Detector and make the changes to one of those datasets using a writer. The process works fine, but it takes roughly a second per records updated. Is that a normal speed? It's alright for small numbers of changes, but this dataset can have thousands of changes from day to day, which would make the workflow run for over an hour, and that's only one dataset when we need to update dozens of them every night. I've looked at the performance tuning article and it either doesn't seem applicable or I don't know enough about the software to know how to check it. My data isn't versioned so I'm using a Transactions writer; I saw another question that had this problem but the answer was to not using the versioning writer. The writer is set to fme_db_operation (so that the updated, inserted, and deleted ports of the Change Detector will work) with Use Existing as the table handling. If there's other inf

1500
kimo
Contributor
jlw_tyrens
Contributor
jlw_tyrensContributor
asked in Data
Corrupt image file when reading and downloading from ArcGIS Portal

Hi all!I have a workflow when I read and download image attachments from ArcGIS Online. I have used the AttributeFileWriter to download the image to a unique filename and it has always worked smoothley.  Now I am trying to do the same with image attachments from our ArcGISPortal and it works fine to read and download the files, but when I try to open the files it says that “It looks like we don’t support this file format.” If I go to ArcGISPortal and manually downloads the image file, it works fine. Does anyone recognize this phenomenon? Best regardsJohan 

520
jlw_tyrens
Contributor
cdejacob
Contributor
cdejacobContributor
asked in Data
problems connecting to an OData feed

I've been trying to access some internal data which is being published from TopDesk via an OData feed. This is the first time I've tried to use odata... I'm using the OData reader, and parameters provided by the TopDesk administrator here. I have a url, a username, and password. I'm using basic authentication. After clicking the browse button beside Feeds, I get the error message "Failed to retrieve feature types."  Does anyone have suggestions on what I can do to successfully connect? Note I also get an error when trying to connect to it from Excel 2016:This sounds like a problem with the account.However, I am able to connect to this odata feed successfully using power BI, and using the same url and credentials as above. During the connection process there, the only difference in dialogues that I can see allows me to choose the "level" at which I wish to apply the login settings to (I chose the root portion of the url).This then lets me view t

2270
redgeographics
Celebrity
spiderman
Contributor
spidermanContributor
asked in Data
How to change the font size of text ,read from DGN File

i have used DGN Reader to read DGN File and it reads great but to modify the font size that read from this file , i use attribute creator and change the value in igds_text_size but i have no effect . could someone fix my problem , why i got no effect of the font size ? and is there other method that i could use to modify the Font size of text ? FME 2020 Thanks in advance

450
virtualcitymatt
Celebrity
harmen
Contributor
harmenContributor
asked in Data
Convert from GeoTIFF to ECW while preserving colors

When I convert a GeoTIFF to ECW, the bright colors of the GeoTIFF are not preserved in the ECW output. Is that possible anyway, to convert it and preserve 100% of the right colors? Searching the knowledgebase, I have found some directions to a solution: How to preserve exact colors in raster conversion .ecw to .tiff? (safe.com) But in the end, how could it be done? Does someone has experience with it? Find examples below of the original GeoTIFF and the ECW output.the faded example is the ECW filethe bright colored example is the GeoTIFF file

1280
danminneyatsaf
Safer
S
sarah.coathup
asked in Data
How to connect to a Progress OpenEdge V11 database

I have created a 32 bit DSN ODBC connection using the progress/openedge 11.7 driver which connects successfully. I am trying to then connect via FME 2020.2 and choosing the ODBC option to connect but I am getting a message returned as follows?The specified DSN contains an architecture mismatch between the Driver and Application 

920
danminneyatsaf
Safer
V
vandijkjoostjan
asked in Data
Exel reader schema difference between configuration and output

When I load my excel file I want the Day attribute to be read as type char so I can split it with string functions (Left/Right). The problem I have is that FME insists on reading the attribute as a 64bit real. I have tried removing the reader, purging the temporary files, saving the workspace, closing FME, and then add the reader again. But I keep having the same result. Are there any suggestions or is this standard for the Exel reader?

120
danminneyatsaf
Safer
G
gnash
asked in Data
Columns in excel (.xlsx) document created using feature writer do not sort correctly.

My workflow reads an SDE, carries out some transformations, and then writes to .xlsx using a feature writer and a template file. When I open the excel file and sort columns the sort function works the first few times but then it begins to only sort certain columns leading to mismatched data in the document. There doesn't seem to be any pattern as to which rows sort and which don't. Sometimes it is only the ObjectID column which will sort.

430
M
chrisgrandin
Contributor
chrisgrandinContributor
asked in Data
Hi, I'm trying to write mesh files incl textures from obj to Esri SLPK format. The problem I face it is takes a really long time. I'm a newbie on the slpk format and don't know if there is a smarter way to do the translation. Any help highly apprecia

Hi, I'm trying to write mesh files incl textures from obj to Esri SLPK format. The problem I face it is takes a really long time. I'm a newbie on the slpk format and don't know if there is a smarter way to do the translation. Any help highly appreciated.

960
chrisgrandin
Contributor
erido
Participant
eridoParticipant
asked in Data
ECW Format v3 Writer error with coordinate system EPSG:2056 / EPSG:21780

With ECW v3 Writer I get the following error message: ECW writer: An invalid parameter was used : date and projection setting are invalid. The coordinate system used is EPSG:2056/EPSG:21780. With the coordinate systems UTM / LL-WGS84 the ECW v3 Writer works correctly!Tested with FME 2019.2.1.0 (Build 19813) WIN64 and FME 2020 Beta (Build 20153) WIN64.

204
ekkischeffler
Contributor
V
vbe
asked in Data
DWG (topographic plan + underground network) to IFC model

(English translation below) Bonjour, Dans le cadre d'un projet, j'ai pour objectif de créer une routine permettant de convertir des plans topographiques et de réseaux enterrés en modèle 3D au format IFC. J'ai commencer à chercher, mais il semble y avoir à la fois beaucoup de possibilités et à la fois aucune que j'arrive à faire fonctionner. Je vous explique un peu mes données d'entrées. J'ai : Plan topographique, dessiné en 2D, contenant des points topographique avec attribut (altitude).Plan de réseaux enterrés avec des étiquettes (bloc ou texte) contenant des attributs divers (nature du réseaux, profondeur du réseau, type d'élément (fourreaux, câbles, canalisation...), altitude de la génératrice supérieur, ect.). Nuage de point j'ai donc plusieurs besoin : 1/ Il faut que j'arrive à faire que mes polylignes représentant mes réseaux (polyligne 2D - Z=0) remontent à la hauteur où elles sont vraiment (soit hauteur TN - Profondeur ou Z de la génératrice

720
V
actualabhijeet
Contributor
actualabhijeetContributor
asked in Data
How to Migrate Oracle Geodatabase (sde) to MySQL Database - ( MySQL FGDB Schema extracted from Oracle database in ArcGIS Pro) in FME Desktop. Which all possible Transformers can be used to migrate this data.

Firstly, we have extracted the Schema of this Oracle database (sde) in MySQL FGDB format in ArcGIS Pro exporting this schema.Now, in FME Desktop I need to connect to Oracle database server and get this data as per feature classes present in datasets with all the attribution and its values with geometry in this MySQL FGDB Schema imported.Need help on this and which transformers can be used to get this Oracle data migrated in MySQL database Schema FGDB.  

380
redgeographics
Celebrity
N
nigellester
asked in Data
Hi All, I am new to FME, I need to convert MapInfo TAB to DWG. I have a polygon file that has attributes and I want to write those polygons and attributes to the DWG file.

The attributes should be visible in the AutoCAD properties dialog.I have been able to write "extended attributes" that you can list with the XDLIST command in Autocad but I want to write attributes that are visible in the properties dialog in Autocad.   Can anyone provide some advice on this or a sample. I think it must be simple but I am missing something???!

950
hrvoje
Contributor
john.beck
Contributor
john.beckContributor
asked in Data
landxml writer bug?

I am doing a simple landxml read to delete unnecessary data from a landxml pipe file and then write it again with the landxml writer. The Struct feature type should write out Invert{}.flowDir, but it doesn't. Anybody have an idea what the problem is?

130
john.beck
Contributor
paulbrandtkcs
Participant
paulbrandtkcsParticipant
asked in Data
How to connect to Azure SQL Database from FME Desktop by using Azure Active Directory Password authentication

When trying to use the SQL Azure SQL Database (JDBC) database connection, can that be used with the ActiveDirectoryPassword authentication type? I have a service account in the Azure Active Directory, but that is not the user our FME Server runs with, to be able to use the ActiveDirectoryIntegrated option. I have successfully used FME Desktop with my personal account as well as the service account (using RunAs) with the ActiveDirectoryIntegrated authentication. Any help you can provide would be very much appreciated.

4540
paulbrandtkcs
Participant
spiderman
Contributor
spidermanContributor
asked in Data
How to get center point of rectangular or square

How to get center point of rectangular or square ?i need to extract the coordinates of center point from any shapeas shown as pic .FME 2020Thanks in advance

380
hkingsbury
Celebrity
deanhowell
Influencer
deanhowellInfluencer
asked in Data
Grid Column to Letter

I have a layer that was created with the tiler transformer and rather than column (0 - n) I want to covert them to an alpha equivalent i.e. 0 => A, 25 => Z, 26 => AA etc Does anyone have a approach for this please?

60
deanhowell
Influencer
A
a_g
asked in Data
Hello! I'm new to FME and I'm wondering if it is possible to integrate PDFs into a CityGML model (LOD1). I need to link a different PDF for each building of the 3D model. Can anybody help me? Thanks in advance

Hello! I'm new to FME and I'm wondering if it is possible to integrate PDFs into a CityGML model (LOD1). I need to link a different PDF for each building of the 3D model. Can anybody help me? Thanks in advance

160
gabriel_hirsch
Contributor
C
cambrien
asked in Data
Why am I missing one record from the output of a Dynamic Shapefile Writer?

I have the need to utilise a dynamic writer for coordinate conversion of a user uploaded file.The schema will be unknown therefore the need to be dynamic.I am using FME (20201027 - Build 20787)I have used a writer transformer and have set the schema to Dynamic and the schema is set as "Schema From Schema Feature".  I am wondering if this has an impact on the first feature being used to dimension the dataset and is not written to the table. However this would be unusual because I have used these writer transformers before with more success. The noted difference I can surmise would be that the schema and features have been split up after the reader and I have used the schema to determine the coordinate system of the incoming dataset features to check that it conforms to the eventual conversion, then the features and schema are brought back together using a Joiner transformer.This works for other formats (CSV and TAB) with which I have used the same logical design.The cached rec

670
C
joy
Enthusiast
joyEnthusiast
asked in Data
FME Desktop 2020: How to reduce DWG file size when converting from DGN to DWG?

Currently, within FME desktop me and my colleague, we are trying to convert a DGN file to a DWG file (realdwg). The DGN file has a size of roughly 90 MB. However just using simply a Reader and a Writer does result in a DWG of around 600 MB! I don't understand why the file became so big and ideally we would like to keep the size of the DWG lower than 100 MB.  Edit:Are there things which we can try to make the filesize of the DWG as small as possible and ideally below 100MB

670
joy
Enthusiast
bernarddinon
bernarddinon
asked in Data
Is it possible in a dwg file to extract value from custom part in BLOCK_DYNAMIC with CAD Readers?

Hello,We are drawing with Autocad. We want to use dynamic block with Custom part (Visibilité1 and Angle1 see PrintScreen 1). With FMe Desktop, is-it possible to have these values in a fields/attribute ? Our objectifs is to import data in a GIS with the values of Visibility to define some objects. Best Regards.Bernard Dinon Print Screen 1

170
bernarddinon
jamal
jamal
asked in Data
FME Desktop 2022: How to dissolve based on a particular field and then to get the COUNT of the dissolved features as well as the FIRST value of other field?

FME Desktop 2022: How to dissolve based on a particular field and then to get the COUNT of the dissolved features as well as the FIRST value of other field? For example, in the screenshot below, I wanted to dissolve based on the “autocad_color” field and then to COUNT the features that have been dissolved while taking the FIRST value of the “landuse_arabic” field. This can be done easily in Pro with one command (dissolve) The data is attached           The data is attached

5430
jamal
jamal
jamal
asked in Data
FME Desktop 2022: How to filter all records based on the “null” value of particular field?

FME Desktop 2022: How to filter all records based on the “null” value of particular field? In the screenshot, I couldn’t figure out how to filter all records that have “null” values in the “ump_layers”    

8220
jamal

Badge Winners

  • FME Form Basic Training
    mmccoshamhas earned the badge FME Form Basic Training
  • FME Certified Professional
    michaelaphas earned the badge FME Certified Professional
  • Welcome Back
    andreashaeslerhas earned the badge Welcome Back
  • Integrate Spatial Data
    cfrenchhas earned the badge Integrate Spatial Data
  • Student (Grey)
    atdvfhas earned the badge Student (Grey)
Show all badges

Community Stats

31,796
Posts
120,781
Replies
39,410
Members

Latest FME

Terms & ConditionsCookie settings
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.