Skip to main content
  • EN

Data

Using readers and writers to extract and load data.

  • 7,082 Posts
  • 26,080 Replies
7082 Posts
mr_fme
Enthusiast
mr_fmeEnthusiast
asked in Data
Oracle Reader in FME Linux

On the safe software site, I realize that there are only 64-bit FME products for Linux. I have a 32bit oracle database and obviously use a 32bit client. How will this linux FME enable the Oracle reader?

70
david_r
Celebrity
fbastian73
Contributor
fbastian73Contributor
asked in Data
How do I read a .log file from a directory and transform it into an excel file?

Hello FMErs,I am wanting to read a .log file from a specific directory, run it pass a tester on a specific attribute and then output the result in excel file format. I plan to place this on the server for self serve.example file attached. I had to change the file extension to .txt due to attach rules.suggestions?thanks in advance.

1180
redgeographics
Celebrity
tim_wood
Contributor
tim_woodContributor
asked in Data
Can I force a translation to fail if violation of a Primary Key occurs when writing to SQL Server?

I'm calling a child Workspace using a WorkspaceRunner. The child Workspace says translation was successful as does the WorkspaceRunner, but when I reviewed the log for the child Workspace, I noticed there were warnings like this: "Provider error `(-2147217873) Violation of PRIMARY KEY constraint 'xxxx'. Cannot insert duplicate key in object 'yyyy'". If this happens, I want the Workspace to fail/terminate. Is there a way I can do this? Would a FeatureWriter transformer rather than a normal Writer give me this functionality?

7924
david_r
Celebrity
T
tjpollard
asked in Data
FME Cloud Workflow for moving media ZIP files between S3 buckets.

Hi Community,After reading this article here, I felt I should tailor my question for my particular challenge and request help from @GerhardAtSafe and @stewartharper, (and if you could point me to the youtube videos and github repo mentioned there I'd be so grateful!) My question pertains to creating a similar process as the "FME Cloud API return ZIP file", but with a few variations in the workflow.What I want to do is call a varying number of CSV files, created as output from another set of FME workspaces executed on Cloud/server, into a workspace(s) which parses some media UUIDs from a couple columns in the CSV into a single line item list. This list of media UUIDs is then used to pull the media from an S3 bucket, and push to another S3 bucket. Now, this next steps is where it gets tricky in the workflow. * How can I enable to ZIP up all the media into one zipped file, give it a date_timestamp file name, such as update_03OCT2018_0830utc.zip, (or what ever may come from s

260
G
londonhb185
Contributor
londonhb185Contributor
asked in Data
Connecting to ArcGIS Online - Unable to apply Token Received from AGO

Hello,I have researched this topic quite a bit and have not been able to solve my issue. I am simply wanting read ArcGIS online feature service layers. I thought it might be a proxy issue, but confirmed with IT that the proxy is fine. I put in my credentials and then click on the ellipses for Feature Service. It will simply look and then time out. I have also tried all the authentication types as well and none of them work.Is anyone else having this issue?

130
nathanatsafe
Safer
M
majobajob
asked in Data
Does JSONTemplater have the ability to create buckets?

Good afternoon all, I'm trying to create a large nested JSON document from SQL to load into MongoDB. Currently I have managed to create the JSON file but i'm not able to load it into Mongo based on the size of the documents. The maximum size for documents is 16MB while my average document size is over 50MB. A solution I have thought of is to bucket each of the sub files so that instead of thousands of documents being embedded into each root there is a maximum limit of say 500 and then a new document is created with the same root, but a continuation of the Sub documents. In the end instead of a single 50MB document I would be left with say 10 identical documents that would hold different Sub documents from one another. I could then query the set of 10 documents using $unwind so that it treated them as a single document of sorts. Does anyone know of a way to achieve this within FME Workbench instead of loading all sub documents into a root?

70
M
M
massimoscocco
asked in Data
Updating CSV file each time a Workspace is run

I have a workspace which reads a CSV (CSV Reader) file for a parameter which is used in a web service call. At the end of the workspace, the CSV file is updated (CSV Writer). This approach work OK on fme desktop; however, when it is run on FME server 2017, instead of updating the CSV file (same file as reader), it simply returns the CSV file as a download--leaving the original file unmodified.Is there any way to ensure that the file which is consumed by the reader is updated by the writer?Thanks

460
mark2atsafe
Safer
lifalin2016
Supporter
lifalin2016Supporter
asked in Data
Limits of PNGRASTER size ?

Hi.I've just tried to create a large PNG raster mosaic. I get this error in the log file:PNGRASTER writer: libpng: Image width exceeds user limit in IHDRPNGRASTER writer: libpng: Image height exceeds user limit in IHDRPNGRASTER writer: libpng: Invalid IHDR dataPNGRASTER writer: Failed to write dataset ....The tile grid is 6 tiles wide and 12 tiles high, and each tile is 20000x20000 pixels.But what is the effective limit of a PNGRASTER output ?According to the libpng documentation, PNG allows 2^31-1 rows and columns, but libpng has "arbitrarily" set the maximum to 1 million. And that's well below my max 240K requirement.Cheers

540
mark2atsafe
Safer
runneals
Supporter
runnealsSupporter
asked in Data
Getting Various 10.6.1 SDE/Oracle errors on truncate/reload

Immediately following the TRUNCATE TABLE SchemaName.TableName SQL command, we have been getting the following:Error executing SQL: 'Underlying DBMS error [ORA-29859: error occurred in the execution of ODCIINDEXTRUNCATE routineORA-00942: table or view does not existORA-06512: at "SDE.ST_DOMAIN_METHODS", line 2352]'Then in another instance (on a separate table) we got this error after it truncated, but on the reloading data:An error occurred while attempting to insert a row into the table 'SchemaName.TableName' using a cursor. The error number from ArcObjects is: '-2147216072'. The error message from ArcObjects is: {Underlying DBMS error [ORA-29861: domain index is marked LOADING/FAILED/UNUSABLE] [SchemaName.TableName]}In both cases, the data was completely truncated, and we copied data back over from our production server. It still occasionally pops these errors up, but other times I can write just fine. We have been having this issue with more than just these 2 tables as we heavily use

340
runneals
Supporter
johnglick
Contributor
johnglickContributor
asked in Data
XML Elements as Attribute Names

I have xml data (see attached). I'm trying to get the following result from this data:ID= 9328Buffer PH = 6.39BS_CA = 68.2and so on. I have no issues flattening the data, my problem is that the "buffer PH" for example needs to be an column with value of 6.39 as the data. But with the way the xml is set up the basic flattening gives me a feature for every <Data/> tag. I can use the transposer, but then I lose other info related to that sample. This particular xml only has one sample, but there can be multiple samples.

212
johnglick
Contributor
J
jamess
asked in Data
Reading DGN 3D solids

I have been given a dgn file and imodel i.dgn that contain 3D solids and shapes. I can work with either file, but need to read the data into FME as 3D solids and translate them to a different 3D data format e.g. FBX. I have had the most luck with the .dgn file using the Bentley Microstation Design (V8) reader with some data read in as a fme_solid, but most of the data has a fme_gemoetry type of fme_line. Previously I had some success reading an i.dgn file using the Bentley i-model Interchange Format reader, but with this data it doesn't seem to read all the 3D objects. How can I read all the data from the dgn files as fme_solids? In Bentley View the model looks like this...In FME inspector using the Bentley Microstation Design (V8) reader it looks like this...In FME inspector using the Bentley i-model Interchange Format reader it looks like this...Note I'm using FME version 2018.1 and have 2019.0 beta installed

650
owen
lifalin2016
Supporter
lifalin2016Supporter
asked in Data
Updating MS/SQL woes and case super-sensitivity ?

Hi.I know that FME is case sensitive per se, but I've just had it report some curious errors in an UPDATE operation.Apparently 5 fields with similar names were spelled differently case-wise in a database that's case-insensitive. E.g. one field was "ImageFile", another "Imagefile1". Renaming the attributes fixed that of course.Before that, it erred because I attempted to update a field with type "varchar(250)" with a value (created in AttributeManager) that FME determined to be "varchar(200)". Yes, really.And lastly, FME erred because there were extra attributes in the feature, that aren't in the database table. Instead of just ignoring them, which really ought to be an option. Removing them solved this issue.Updating a larger field with a smaller value should also just be accepted, and not throw an error.And FME already prompts the target database for all sorts of information before pushing data into it. Would it then be too much to a

260
lifalin2016
Supporter
S
san
asked in Data
Errors while converting obj to gltf

conversion log:OBJ Reader: Token Unsupported: 'Ni' - IgnoredOBJ Reader: continue reading with next good featureOBJ Reader: Token Unsupported: 'illum' - IgnoredOBJ Reader: continue reading with next good featureFailed to read texture: 'C:\\test\\collada_data\\Sedan_W3_Destroyed\\sedan\\sedan_metro_orng_pan_burn_w3_d.tga'OBJ Reader: Token Unsupported: 'Ni' - IgnoredOBJ Reader: continue reading with next good featureOBJ Reader: Token Unsupported: 'illum' - IgnoredOBJ Reader: continue reading with next good featureFailed to read texture: 'C:\\test\\collada_data\\Sedan_W3_Destroyed\\sedan\\sedan_bottom_w3_burn_d.tga'OBJ Reader: Token Unsupported: 'Ni' - IgnoredOBJ Reader: continue reading with next good featureOBJ Reader: Token Unsupported: 'illum' - IgnoredOBJ Reader: continue reading with next good featureFailed to read texture: 'C:\\test\\collada_data\\Sedan_W3_Destroyed\\sedan\\sedan_w3_decals_d.tga'OBJ Reader: Toke

550
geosander
T
tgolden
asked in Data
I am trying to create an xml reader. I continue to get the errors listed below.

I am trying to create a proof of concept. Reading a web service and then writing to a sql server db. I am using the XML format for the reader. Getting the error message below:  Starting translation...  FME 2018.0.1.1 (20180615 - Build 18312 - WIN64)  FME Home Edition (node locked-crc)  Temporary License: 265 days left.  Machine host name is: 8BCCFH2-LT  Operating System: Microsoft Windows 10 64-bit (Build 16299)  Copyright (c) 1994 - 2018, Safe Software Inc.  Safe Software Inc. Potentially retrieving dataset 'https://www.applitrack.com/Littleton/api/AppliTrackAPI_v3.svc/Universities()' for reader  Retrieved file C:\\Users\\tgolden\\AppData\\Local\\Temp\\FME_TempFileStore_1538683504282_11704\\Universities() for reader  Creating reader for format: XML (Extensible Markup Language) Trying to find a DYNAMIC plugin for reader named `XML' Loa

2300
takashi
Celebrity
D
dmyers891258
asked in Data
PDF Reporting of updates made to hosted feature service on AGOL

Is it possible to create an FME workflow that will periodically check a hosted feature service for updates or edits, and then generate a report of those edits made and by who? If someone has done something similar before please let me know.

80
D
oliver.morris
Contributor
oliver.morrisContributor
asked in Data
MS SQL Server (JDBC) Writer: Error writing feature. Error: serializeGeometry not implemented

Has anyone experienced a similar warning/error in FME when using the SQL Server JDBC writer, writing out to a binarygeometry data type?If so am I ok to ignore, it only came up as a warning not an error.Many ThanksExample below:MS SQL Server (JDBC) Writer: Error writing feature.  Error: serializeGeometry not implemented+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++Feature Type: `dbo.V_PLAYFINDER_PXS_LOCATION'Attribute(encoded: UTF-8)     : `LONG_ABBREVIATION' has value `CAM'Attribute(encoded: UTF-8)     : `MID_X' has value `-60.59640903'Attribute(encoded: UTF-8)     : `MID_Y' has value `13.31999124'Attribute(32 bit integer)     : `OBJECTID' has value `106'Attribute(encoded: fme-binary): `POLYLINE' binary value as HEX `010200000002000000CC323298041250C0C03E48C3E7602B40C03AC512A5744CC060EE7D05C4E62940'Attribute(encoded: UTF-8)     : `PRODUCT' has value `PC_G2020X'Attribute(32 bit integer)     : `PRODUCT_ID' has value `1894'Attribute(encoded: UTF-8)     : `PROD

350
oliver.morris
Contributor
K
kam
asked in Data
Resources on writing features to databases

Hi FMEer's!Can someone kindly point me to tutorials/webinars of where the workbench is writing particularly large amounts of data to spatial and non-spatial databases? I have many models that write to Oracle databases, SQL Server, ArcSDE etc. However, I find that the moment we start writing 100's of thousands of records it is quite slow. It's generally quicker to import a table into one of the databases and do the truncate and load through a database management software. Some of the methods I use in FME are: - Change detection and use FME_DB_OPERATION = Insert or Delete (Not sure really how to use the Update) - Truncate table and insert (Use this when the table is massive and change detection is just not feasible)- Write to ESRI file geodatabase first and then copy to ArcSDEAny help is much appreciated, I just want to make my models more efficient! Kind regards,Kam

90
mark2atsafe
Safer
G
gidolangen
asked in Data
Is it possible to add items to the FMW script?

I have an FMW script that includes an AttributeCreator transformer. I want to include 500 additional attribute creator transformers. It would be tedious to create them in the workbench. Instead, I would like to create them with a little Python script and then splice the generated code into the FMW script. Is there any documentation on how to manipulate the FMW text file?

410
G
M
majobajob
asked in Data
MongoDB GridFS - Huge Document Sizes

Does anyone know if FME has compatibility with MongoDB's GridFS to store documents that are larger than 16MB? I've written an embedded JSON but the average document size is 49MB. Will I need to load it into mongo outside of FME or is there capability for this currently?

90
M
V
venkenk
asked in Data
PTS to LAS problems

Hey all,I'm a new user of FME. And we are using FME (or trying to) to convert bathymtrie data (in PTS format) to geotiffs. We used to do it in C3D but after seeing some vids of fme i think it will be allot faster and easier using fme.So i googled it and tried some things but i keep ending up with the same error log ( in a picture below.). After some more search work i got the advice to first convert the PTS format to a LAS format. But still i'm givin the same error. I'm guessing there is something wrong with my parameters? Thanks for the help. Grts,Koen

450
H
majkiboy
Contributor
majkiboyContributor
asked in Data
Transpose a table by attribute but retain the rest

This is what I haveThis is what I needAs you can see I need to transpose the ID but retaining the rest. I am unsure how to phrase the question so I guess the images shows my problem better than my phrasing!My table has more than thousand records and I tried many combinations of sorter, counter, attributeexploader and so on until the point that I am very confused as how to even approach this in FME. Any suggestions are appreciated! And yes, I'm very new to FME. Output should be a csv but that is less of a concern right now.

220
takashi
Celebrity
M
mikefrommaine
asked in Data
Modify and Write attributes with XML Writer

Hello. I have to read a number of xml files that are "version 3", modify the contents and write them out as "version 4". The changes include renamed xml tags, some reorganization, addition of some new tags etc. When I read in an xml it deserializes the contents into attributes (which is really great!). I'd like to make my modifications to these attributes (and their values), then write out the result. Instead it writes back out the original file contents which it has saved in the attribute xml_fragment. Is there a way I can tell the writer to output the contents of the attributes and not the xml_fragment?

140
jdh
Contributor
S
san
asked in Data
FMW file creation in cmd

Is there any possibility to create a fmw file on the fly in the command line itself,with coordinate system and file types source and destination as arguments.

70
david_r
Celebrity
johnglick
Contributor
johnglickContributor
asked in Data
Combine binary pdf files?

I'm receiving binary pdf files from a web service. I've successfully decoded and written them to my file system. However, I would like to combine certain files with a similar attribute into one pdf of multiple pages. Any ideas how I can do this in fme?

590
johnglick
Contributor
A
antoninotb
asked in Data
Filename and path to open from another source (from DB table)

I need to read a filename and path from a database, then use these values to open the file on the File System. Unfortunately the Reader standard do not take filemane from previous features. How may i handle that ? Thanks

60
david_r
Celebrity

Badge Winners

  • FME Certified Professional
    _jacques_has earned the badge FME Certified Professional
  • Integrate Spatial Data
    madihas earned the badge Integrate Spatial Data
  • Webinar Watcher (Bronze)
    bradmakesmapshas earned the badge Webinar Watcher (Bronze)
  • English Speaker
    bradmakesmapshas earned the badge English Speaker
  • English Speaker
    jamatsafehas earned the badge English Speaker
Show all badges

Community Stats

31,790
Posts
120,761
Replies
39,406
Members

Latest FME

Terms & ConditionsCookie settings
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.