Skip to main content

Data

Using readers and writers to extract and load data.

  • 6,973 Posts
  • 25,687 Replies
6973 Posts
hlouie
Contributor
hlouieContributor
asked in Data
SDE Reader Dynamic or Generic

Is there a way of creating a Workspace where I can have a selection list (user parameters) of Feature Type (GIS.WaterMain, GIS.CombinedMain) to read the Schema with attributes (is this dynamic of generic)? I've been able to create a Dynamic Reading but Schema is fixed to attributes in the Reader, Generic is only shows on field.

20
hlouie
Contributor
S
shazadtareen
asked in Data
How to delete/truncate records in a feature class in feature dataset sitting in ArcSDE 10.2.2

@shazad.tareen@gmail.com

2370
david_r
Celebrity
arnovananrooij
Contributor
arnovananrooijContributor
asked in Data
How to convert from dgn faces to fme surfaces?

Hi, I have a dgn with faces which I want to convert to fme surfaces to be able to put texture on them. FME reads these faces form dgn as polygons. I tried to convert these polygons to faces with the FaceReplacer but a lot of the polygons failed because they are not planar. If the FaceReplacer works I plan to use the GeometryCoercer to convert the faces to surfaces. Does anyone have an idea how to solve this problem? Thank you

330
arnovananrooij
Contributor
takashi
Influencer
takashi
asked in Data
Google BigQuery Reading Failure

FME 2019.0 build 19238 Desktop on Windows 10 and Server on Linux (CentOS 7) I'm trying to migrate a large dataset (tens millions records) from a Google BigQuery table into a PostgreSQL table, but the translation failed and this error was logged. Tried some times, the translation stops always with the same error. Translation FAILED with 1 error(s) and 0 warning(s) (11375000 feature(s) output)FME Session Duration: 59 minutes 9.9 seconds. (CPU: 1945.4s user, 53.5s system)END - ProcessID: 11848, peak process memory usage: 1027252 kB, current process memory usage: 843964 kBjava.lang.Exception: Google BigQuery: Error retrieving query results from job '****************:********.********' executed on table job_*_***************. Provider error: 401 Unauthorized{ "code" : 401, "errors" : [ { "domain" : "global", "location" : "Authorization", "locationType" : "header", "message" : "Invalid Credentials", "reason" : "authError" } ], "message" : "Request had invalid authentication credentials. Expe

250
takashi
Influencer
N
nevilleda
asked in Data
FME Desktop - Truncate append

I am running FME Desktop 2018.1 in a Windows 10. My source data is in a SQLServer database (2016 R2) and the destination feature class is in an ESRI Enterprise Geodatabase (ver 10.5.1) also using SQLServer (2016 R2) as the backend database. The destination feature class is in a Enterprise spatial data warehouse, business decisions are make on the contents in the destination feature class. I have a FME Workspace that reads a table with latitude and longitude and writes to a feature class in a ESRI Enterprise Geodatabase. The writer is set to TABLE Exists: 'yes', TRUNCATE Table First: 'yes' It appears that FME: 1) Truncates data in the destination feature class, then 2) Reads the source table, then 3) Writes to the destination feature class. Is this correct or have I misinterpreted the FME log files? If this is correct, then there can be considerable time when the destination feature class is empty or being updated. If FME does truncate the destination data first, then can I change the o

700
dellerbeck
Contributor
T
tsurveyor
asked in Data
Write to Excel template header into specific cells

I have created an Excel template to be used as backup for billing expenses. The invoice has a header with specific places for billing date range(start, end), project name, project number, and project manager. The final output has a separate sheet for each project number. I have added AttributeManager to get rid of all of the other attributes and only have the 5 I need to insert into my template - the itemization is added later. I have everything working except getting the aggregated attribute value (i.e. project name) added to the named cell. How do I specify where to write each attribute? Version 2018.1 Desktop

1040
pratap
Contributor
B
brendenroche
asked in Data
building a heirachical xml file from a flat list

I have a list of code numbers and descriptions. i wish top converted them into an xml file that has parent child relationship. Can any suggest tutorial or content that is appropriate? Thanks B here is a screen shot of my list and here is what i need

360
M
tim_wood
Contributor
tim_woodContributor
asked in Data
Connecting to SQL Server when runnning as a scheduled task

I have a Workspace running in FME Desktop 2017.1.1.1 on a server. The Workspace has a SQL Server Spatial Reader in it. The SQL Server database is on a different server which is in the same domain. The Reader is using a named SQL Server connection which is publicly visible. The connection uses SQL Server credentials to connect to the database (not Windows authentication). The connection is not encrypted. The Workspace runs fine when I'm logged in. It works as a scheduled task when running as me. But when I try to run it as a scheduled task using a different user (we have a Domain service account for running scheduled tasks), it fails. I've tried the registry fix suggested here: https://knowledge.safe.com/articles/32356/scheduling-software-doesnt-recognize-fme-connectio.html However I still get "Invalid authorization specification" in the log file. Interestingly, since applying the above fix, I also get errors about encryption (see below). I don't know if this is progress or not! One thi

3590
D
bjudes
Enthusiast
bjudesEnthusiast
asked in Data
Overlay label from FGDB to Raster

Hi, I want to overlay labels from a FGDB point file on a raster. I'm using the Labeller transformer but my points never pass through. I've uploaded a workbench. Thanks for any help. B

400
bjudes
Enthusiast
sameer
Contributor
sameerContributor
asked in Data
InterBase & Ingress

Hello, Main Query: I need to connect to an InterBase DB and it's not listed in the "Reader Gallery".I have installed the ODBC driver(s) for it. Can someone please advise how to go about creating this database connection possibly using the FDS file import? What about making use of an ODBC connection? I can already connect to this database using the IBProvider driver on a different server using a different application. In this particular case though, I must use FME to connect to this table to build a test Corporate Notification Service.Secondary Queries: I also noticed that Ingress was not listed as a "Reader Format" by default. Do you have any plan for implementing this? Has anyone managed to use FirmStep APIs in FME? Thanks, Sameer

250
sameer
Contributor
R
robclark984
asked in Data
Writer ArcGISPortalFeatureService Token Management - FME Desktop

In some of my workbenches in FME Desktop I can have over 400k of rows to update to a ArcGIS hosted portal feature service. However the writer will fail somewhere between 400k to 450k, roughly after an hour of updating. I think this is the token from portal expiring, but can't find a way to change the expire duration in FME Desktop when it gets a token from portal. Is there a way around this?

110
runneals
Contributor
mmestrov
Contributor
mmestrovContributor
asked in Data
Determine if a Writer which is an SDE layer is versioned?

I need to figure out a way of testing if a writer which is an SDE layer is verisioned or not. If it is versioned I do not want to truncate and fill if from the reader but if it is not versioned then I would want to truncate and fill.

50
erik_jan
Contributor
F
fme_fan_mike
asked in Data
Can't Add a New .csv Reader after Installing 2019.0

I'm sure I'm overlooking something embarrassingly simple but after adding 2019.0 to my computer (currently also have 2018.1 installed to a different path) I cannot complete addition of a .csv reader. I'm able to preview the input files and they show the correct column names and content, but the 'ok' button is greyed out. . preventing me from ultimately adding the reader. Automatic and Manual attribute definition give the same result. I don't have this issue with the Excel reader. Thanks in advance - pulling my hair out!

110
danilo_fme
Evangelist
tsirkuse
tsirkuse
asked in Data
ExcelStyler - Conditional background formatting based on value of another cell

I would like to color the cells of my first column ID_WS based on the text value contained in a cell UPDATE_STATUS which basically contains the fme_db_operation value. I tried to apply the styling with ExcellFormater but it doesn't seem to be working in the output. What am I doing wrong here ? Thank you !

1100
ebygomm
Influencer
checcosisani
Contributor
checcosisaniContributor
asked in Data
Extract info from WEBsite

Hi I need to extract info from XML I used the HTPP caller but now I need to extract the string related to tag <tbody> (see picture) and I don't know how can procede thx Francesco

50
checcosisani
Contributor
T
tobiaustaaibos
asked in Data
How can I set values for attribute igds_model_name when writing to DGN file?

Hey, I am dealing with a data migration from File Geodatabase to Microstation DGN right now. I am wondering whether it is possible to set the value for the attribute igds_model_name in the DGN Feature Writer. The only accesible attribute is igds_xref_model_name. But I need igds_model_name, because right now all written features get the value Default, which is wrong. The values I want to write are RB, GEM and DIG. They are currently saved in an attribute in a File GGeodatabase.

264
T
P
pedrosegeom
asked in Data
Postgis 2.1

I have a workspace to convert shapes files to Postgis 1.5 working fine In the same workspace, I change the conection parameters to conect the same table in Postgis 2.1 but I get an error because of Srid restriction. Reading the log, I see is trying to write SRID=1 (the postgis restriction is to Srid=25830, in both versions of Postgis, so it returns an error) The conection parameters are the same in both conections If I inspect the entity that its trying to write, the properties gives EPSG:25830 Coordinate System, but in the log writes SRID=1 Why is trying to load Srid=1 when I change conection to Postgis 2.1? Thanks

123
rahulsharma
Safer
G
gauripearl
asked in Data
Can I combine two KMZ files and extract the result of combined kmz's in a csv file or a table with FME?

I have two KMZ files containing states and cities borders. I want to combine them and find out which city falls under which state as a result in some tabular format. How I can achieve this with FME? Are there any transformers available for this?

250
chrisatsafe
Contributor
jbradfor
Contributor
jbradforContributor
asked in Data
S3 GeoJSON Reader

I do not see a reader yet that can read a GeoJSON file in S3. Rather than needing to download it via the S3 transformer. We would like to use S3 as a spatial data hub so it can be used by many tools.

130
jbradfor
Contributor
fikusas
Contributor
fikusasContributor
asked in Data
Dynamically read WFS with token generation

I'm using FeatureReader with following parameters: Dataset: $(Source_WFS:) Feature Types To Read – From Attribute: _FETURE_TYPES $(Source_WFS:) is composed of some fixed URL address (below is an example one, not the real one I'm using) plus dynamically generated token by Python script in private parameter called Token which is: import fmeobjectsimport urllibimport urllib2import json username = FME_MacroValues['Username']password = FME_MacroValues['Password']client = FME_MacroValues['Client']expiration = FME_MacroValues['Expiration']format = FME_MacroValues['Format']url = 'https://www.example.com/tokengen/' values = {'username': username, 'password': password, 'client': client, 'expiration': expiration, 'f': format}data = urllib.urlencode(values)request = urllib2.Request(url, data)response = urllib2.urlopen(request).read()jsonToken = json.loads(response)return jsonToken['token'] This worklfow works perfecty in FME 2014, but in FME 2018 I get the error, when trying to load features to Ou

310
xiaomengatsafe
Safer
mukhtarg
Contributor
mukhtargContributor
asked in Data
shp to excel

Hi. I want to export some fields from shape file to excel. Any help in this regard would be highly appreciated.

470
david_r
Celebrity
candemiroguz
candemiroguz
asked in Data
DGN to FME Geometry Problem

Hi all, I've 3D model of building sample as you can see at the .jpg file.I worked in the .dgn file.But when i import .dgn file to FME data inspector i've more geometry about drawing.I think that FME is handing logs .dgn file geometry history cause i copied exactly same coordinate into building room drawing geometry. So, how can i resolve this problem? Can i resolve in FME Desktop with transformers? (MS Drawing) (Same .dgn file looks in FME)

430
candemiroguz
dbklingdom
Contributor
dbklingdomContributor
asked in Data
Extracting HTML from the PopupInfo field of a GIS Layer

Hello, What we do is get KMZs and run the ArcGIS tool to convert them to to layers in a file GDB. When you use the layer in ArcMap there is a field PopupInfo, which has HTML data we need to extract?? I have tried several recommendation with HTML, but nothing gets me to where I can flatten the data out into a usable table. Please let me know any suggestions?

4077
dbklingdom
Contributor
dallalana91
dallalana91
asked in Data
Do Civil 3D Survey Figures are supported by FME 2018 or FME 2019??

Hi there... Sorry about my English. I'm trying to translate survey drawings made in Autodesk Civil 3D 2017 and 2019 to GDB using CoGoPoints and Survey Figures. For CoGoPoints no problems at all, but Survey Figures are not supported at my FME ESRI Desktop 2017.0.0.1. If the Survey Figure is supported by FME Desktop 2018 or newest, I will ask to my company for a software update, if not I need change my strategy. Thanks in advance! Demetrius

60
dallalana91
R
ramotswa
asked in Data
Parsing XML properties properly

Hello - part 2 of an XML thread. I have a bunch of nodes in an XML document that look like this; <nodes> <node> <properties> <property type="field_name">first_name</property> <property type="field_data">Ben</property> </properties> </node> <node> <properties> <property type="field_name">last_name</property> <property type="field_data">Potts</property> </properties> </node></nodes> And I want to get those out as variables to use down the line in an FME workbench. At the moment I am relying on the order of them in a Test Filter (i.e. If @Value(_attr_name) = nodes.node{0}.properties.property{1} = first_name then put it on an output port of 'first_name') - however I hate relying on the order and I'm sure there is a better way of getting them out. Any help would be much appreciated. Ramo

111
takashi
Influencer

Helpful Members This Week

Badge Winners

Show all badges

Popular Tags

Community Stats

31,148
Posts
118,077
Replies
38,765
Members

Latest FME

Terms & ConditionsCookie settings

Cookie policy

We use cookies to enhance and personalize your experience. If you accept you agree to our full cookie policy. Learn more about our cookies.

 
Cookie settings
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.