Skip to main content
  • EN

Data

Using readers and writers to extract and load data.

  • 7,090 Posts
  • 26,100 Replies
7090 Posts
oliver.morris
Contributor
oliver.morrisContributor
asked in Data
FME Cloud writing from postgis to fgdb -Geodatabase Error (-2147220987): The user does not have permission to execute the operation.

Hi, we are coming up against an issue now that is now a real blocker to our workflow. We are running a workbench attached on FME cloud which picks up content in postgis and writes to fgdb. The problem is at random - (honestly) it will fail with the error:Geodatabase Error (-2147220987): The user does not have permission to execute the operation.If you run the task again specifically running on the feature it failed on, it will run no problem.I know there are other cases of this reported, however when we run locally there is no issue.https://knowledge.safe.com/questions/45200/writing-esri-file-geodatabase-issue.html https://knowledge.safe.com/articles/362/workspace-fails-on-fme-server-when-using-unc-paths.html @brianatsafe @david_r @mark2atsafe have all commented on this error before, reading through I can't see any issues that relate - as we are running as superuser and also have tested locally.The two logs attached outline the error, the larger log, is for running a very big task

130
oliver.morris
Contributor
S
sara_lenakordas
asked in Data
Reproject Point Cloud

Hello! I have to transform a PointCloud Dataset (LAS) from EPSG 31256 to EPSG 4326. Therefore I used the transformer CSMapReprojector. My output PointCloud always looks like this: This was the Input dataset:  Thanks for your suggestions.    

1960
takashi
Celebrity
B
bowlesg
asked in Data
Esri ArcGIS Portal Reader

I am trying to connect to a hosted feature service in our GIS Portal. I am receiving the following error:..."Check your credentials and try again. The error code from the server was '400' and the message was: 'Unable to generate token.' A fatal error has occurred. Check the logfile above for details"...I am using our fully qualified domain name with /sharing/rest/generateToken with our service account credentials.Ideas??Thanks,--gary

1170
pingotech
Participant
fmelizard
Safer
fmelizardSafer
asked in Data
What is a Point Cloud? What is LiDAR?

I Want to know More aboutWhat is a Point Cloud? What is LiDAR?  I Need Brief Clarification i have Seminar on this topic. your Answers would be appreciated.

110
jovitaatsafe
Safer
sunsilk11
Contributor
sunsilk11Contributor
asked in Data
Combine XML and SQLCreator to process records

@takashiHi I am using an SQLCreator to return all incidents reported in the last hour. I then want to use an XML file to define levels for each alarmtype, which determine whether an alert record is created depending on the type of incident and count e.g. <incidentype name="gasleak" level="1,5,10"/><incidenttype name="electricityinterruption" level="1,3,10"/> therefore for gasleak, an alarm record is created if count of incidents is as follows:count alarmcreated Level AlarmTime 2 Yes 14 No N/A7 Yes 511 Yes 10Note when count=4, no alarm is created as level 1 alarm has already been created.So far, I have added an XML reader and use FeaturePaths. I have also obtained the count of incidents for the last hour using SQLCreator. Any suggestion how FeaturePaths could be used in this case? Is pythonXML the b

69
takashi
Celebrity
B
bram
asked in Data
Is it possible to limit AGOL reader to subset?

I've just started using FME. We have a few feature-services in ArcGIS Online with a lot of features (+/- 150.00). We are creating replica's every night on our server by using a Python script from esri. In essence the script works fine. But the replica's fail at least half the time because of time-outs, or sudden server error's on esri's side. I want to know if we can replace some of our workflows by using FME. I've created a script that reads the AGOL feature service and checks for changes/new features. This gets written to our Postgres database. The script works fine, but again, I get the same errors.I've changed the batch size to 10 features, but no luck. There's an option in the reader to start reading from a certain feature. I thought this might be a solution, but it still starts from 0 (maybe it's not reading / downloading it, but it still runs every 10 features. The corresponding log is: "ArcGIS Online Feature Service Reader: Getting featu

250
B
C
ckintanar
asked in Data
date loaded incorrectly into oracle Date field, works fine in Windows.

Hi I'm loading data from an ESRI .gdb file geodatabase into an Oracle database. When I run the .fmw script from Windows, the date loads fine into an Oracle table 'date' field (e.g. 06/22/2019). But when i run the same .fmw script from Linux FME Desktop, the date in oracle (same oracle db) is messed up (e.g. 02/12/1937). Anyone know how to fix? NOTE: Windows is FME Desktop 2017.1.2.1 Win 32, Linux version = FME 2017.0.1.1 (linux-x64). The .fmw script used was created from the Windows FME Desktop. All other fields seem to load just fine.

300
M
bernarddinon
bernarddinon
asked in Data
How to order the objetcs of different layer with a dwg writer?

Hello,  I want to write a dwg with several layer. In layer i can have block, points or Text. How can i do to order the objects in a systemaic way?  Bernard Dinon

190
bernarddinon
Z
zatom
asked in Data
how to transform multi-line json to one line valid query that can be used through FME transformes:

I can not just copy the curl command from kibana and paste it into the FME transformer. Can you please provide me helpful sites, links, where i can learn how to tone it down to one liner query to use in FME transformers?For example, in kibana console, the following code gives me the documents that matches company_name=shell. But, if I need the same result in FME, i have to use FeatureReaders/httpcaller/elasticReader with the query: GET /prod_gffp_filings/_search?q=COMPANY_NAME:chevron  GET prod_gffp_filings/_search?pretty{ "query": { "match": { "COMPANY_NAME": "Shell" } } }

220
laurawatsafe
Safer
K
karltk
asked in Data
Revit Native Reader does not support transparency

Just using the sample file rac_advanced_sample_project.rvt Even not in data inspector, it does read most of it correctly, but not glass.  Where it should looks like this:  So is there any more features that not currently be supported?

40
daveatsafe
Safer
joshuadamron
Contributor
joshuadamronContributor
asked in Data
RasterReplacer Error: "libjpeg: JPEG datastream contains no image"

Howdy,Two questions: 1) Is there a method for using a Tester like transformer to identify if a "JPEG datastream" contains an image or not prior to a record reaching the RasterReplacer. Details below.2) Purely out of curiosity, how would I go about turning off the Translation Log error referenced in the snippet below? (...this warning can be turned as an error by setting GDAL_ERROR_ON_LIBJPEG_WARNING to TRUE) I'm processing images that were taken via Esri's AGOL Collector as attachments to a feature record. In my workflow I'm using RasterReplacer to extract the images from the AGOL data field (arcgisonline_attachment{0}.data (encoded: fme-binary)) and I have come across an issue where some images are corrupted.When these bad images reach the RasterReplacer it throws a translation error "JPEG reader: libjpeg: JPEG datastream contains no image". Full translation log error message below: 2019-05-14 14:05:48|   6.8|  0.0|WARN  |JPEG reader: libjpeg: Premature end of JPEG file (this warning 

850
joshuadamron
Contributor
D
dewan
asked in Data
Upload the folder in S3

I am working on FME 2019 in windows. I want to upload a folder on S3 bucket but do not want the parent folder to be included. Be like:C:\\abc\\xyz\\mno.zip, C:\\abc\\xyz\\nbc.zip - these zip files to be uploaded toAmazon S3\\abc\\prtand the data should be replaced with the new one(the old data exit in the folder on S3.But, when I say upload the folder, it is like- Amazon S3\\abc\\prt\\xyz\\mno.zip, Amazon S3\\abc\\prt\\xyz\\mno.zipand I want it to be like- Amazon S3\\abc\\prt\\mno.zipSo, basically no parent folder in between. Also, if I select data source to upload to file, I can only upload 1 file, not multiple.Any options to upload multiple files? Or to upload the folder but not including parent folder.Thanks.  

440
takashi
Celebrity
T
tfsrichard
asked in Data
A compatible ArcGIS Python within the supported range could not be found

I have installed ArcGIS Pro 2.3.3 which uses Python 3.6 with FME Desktop 2019.0.1.0 64 bitFME log is returning error in the title with FME Translation Settings:Preferred Python Interpreter:ESRI ArcGIS Pro 2.1/2.2/2.3 Python (3.6)ArcGIS Pro is licensed to work offline.Any suggestions to get this working appreciated.

410
trentatsafe
Safer
oliver.morris
Contributor
oliver.morrisContributor
asked in Data
shapefile delete record - feature writer

Hi, does anyone know of a nice trick to delete features (rows/records) from a shapefile. We currently do this using the delete property in feature writer but this isn't available with shapefile.We basically want to be able to delete where a property = true, the only other thing to mention is this might leave the shapefile empty, so we cant simply use a filter and a writer because FME wont write out an empty shapefile.Many Thanks,Oliver

1040
oliver.morris
Contributor
F
fordy
asked in Data
Alteryx Process

Hi, Im trying to emulate an Alteryx process, but I dont quite know how to do it. Here is the video to show a summary of how to do it https://www.youtube.com/watch?v=nmbWf6-PFhE  And here is the test data  <area shape="poly" alt="" coords="496,172, 496,232, 566,232, 566,121, 496,172" href="BLUE"> The process is taking a shape from adobe illustrator, exporting to HTML (as above) then extracting each of the above into a csv that can be read in tableau and produce a shape.  Ideally from this <area shape="poly" alt="" coords="496,172, 496,232, 566,232, 566,121, 496,172" href="rrrr"> to this BLUE,X,496,Y,172BLUE,X,496,Y,232BLUE,X,566,Y,232BLUE,X,566,Y,121BLUE,X,496,Y,172 There maybe multiple variables in the amount of X and Y coordinates per shape. Anyone got any ideas on how to do this? Thanks Chris

590
jovitaatsafe
Safer
tim_wood
Contributor
tim_woodContributor
asked in Data
Best way to upgrade a Worksapce to the latest version?

I've taken some Workspaces created in 2015/2016 and saved them as 2017.0.1.1 Workspaces. I've removed Readers and Writers and added them in fresh. In some cases I get this report in the log:Workbench: Workspace integrity check during load found one or more readers/writers with configuration issues.Workbench: Backed up current workspace as: E:\\AddressBase\\Plus\\FME\\LACHS (backup).fmwWorkbench: Fixed reader/writer configuration issues.I know this means the issues should be fixed, but it raised the question in my mind about the best way to upgrade Workspaces. I know old ones should generally run in a later version of FME but in these cases I'm changing the Reader(s) and/or Writer(s) so I might as well update the Workspace version.Some Workspaces have a lot of transformers in, so is a good upgrade path to create a blank Workspace, add the Reader(s) and Writer(s) then copy and paste the transformers from the old Workspace (then upgrade transformers and edit parameters if n

1290
gio
Contributor
pratap
Contributor
pratapContributor
asked in Data
SmallWorld - More Options

When I looked at the below linkhttps://knowledge.safe.com/articles/48316/importing-into-smallworld.html I came across the options "Export to FME" and "Import from FME" in Folders dialogue boxDoes anyone used these options (refer image at red polygon)??? If yes could you kindly provide some information about uses/functions/requirements to use themThank youPratap

200
zzupljanin
Contributor
D
davebranson
asked in Data
Does FME support the "Climate FieldView" .DAT file format?

I have receive field application and harvest maps (.dat) from Climate FieldView. I am quite certain these are raster images and they don't appear to be compatible with any of the existing FME formats. I believe they may be of the same format used by SST Summit, MapShots AgStudio, SSI Agvance Mapping, AgIntegrated. Does anyone have experience with reading these files?Ideally, we would like to get them into a format where we could show them as historical reference files (GeoTiff, etc.) within an app that is currently using WKT and MySQL Spatial.

1020
johnglick
Contributor
bruceharold
Supporter
bruceharoldSupporter
asked in Data
Does FME 2019.1 support Sharepoint columns of the Calculated type?

@mark2atsafe I can see Lookup type was added for 2019.1, is calculated there too?

40
bruceharold
Supporter
A
ahotschek
asked in Data
Autodesk Revit Reader - Missing Attributes and Geometries

Hey,I'm using the new Autodesk Revit Reader to import a Revit model. It imports all the objects, but a lot of the used attributes are missing. Also some of the objects, mainly objects of the Architectural.GenericModels have no geometry (which should be solids). I already tried to expose the missing attributes, but I had no luck doing this. Are their any options to get the missing attributes and geometries or do I have to wait for a more developed version of the Autodesk Revit reader?Thank you in Advance!

323
A
W
witos
asked in Data
when fme saves features to temp ffs files

Hello, In my case there a a feature reader, then some tranformators for removing some attributes and adjusting fanout expression and then gml writer.  Why fme saves some partial results to ffs even if there is a lot of free memory? I think that it takes a lot of time causing performance impact.  part of log:2019-05-28 12:07:33|1909.9|  0.0|STATS |Storing feature(s) to FME feature store file `C:\Temp\2\no-conn-ffs-1559036060497_7200_14.ffs'2019-05-28 12:10:00|2055.8|145.9|STATS |Stored 1009731 feature(s) to FME feature store file `C:\Temp\2\no-conn-ffs-1559036060497_7200_14.ffs'

980
david_r
Celebrity
gisgeek
Contributor
gisgeekContributor
asked in Data
Reading XML from a website

Hi I am trying to bring in an XML table from http://envdata.es.govt.nz/services/data.ashx?f=water-level.xml which has some useful water level data. I have tried different XML readers and can't get it to read the data. Can someone recommend a workflow that will read it. 

40
takashi
Celebrity
T
toshibamaher
asked in Data
Construct 3D city model using Laser Scanning

hello every oneMy name is Maher I am student at University of Oulu FinlandNow I am Working on modeling 3D City Model using FME .I am using GML File and Laser Scanning data .I used a workflow and I got good result .However the roofs of All building are considered as a flat surface and the details of the roofs are not taken on consideration 'Because of that I would like to Ask you if you can help Me to construct the full Building with Its roof detailsLas data is in the LINKMy Previous Result LINK

90
jovitaatsafe
Safer
N
nademmy
asked in Data
Wait for SpreadSheet to be updated before reading

I am manipulating a Google Sheet by inserting many rows of data via a Google Script triggered by my FME Workbench. What I wish to do is to read the Google Sheet all over again once all the insertion have been completed. I couldn't think of a way to do this except by pausing the whole workbench while the insertions are being donne.  Is there a more time-efficient way of achieving the same result?

130
chrisatsafe
Contributor
S
saqibamin
asked in Data
Problem opening large MS Excel file in FME

I have a fairly large *.xlsx Excel file (27 MB), when I attempt to open this in an FME Desktop reader, upon selecting the "parameters" button or pressing "OK" the application crashes.If I re save the file as a *.xls file, this removes data and re-saving as an xlsx file doesn't achieve anything either.Any Ideas?

360
fmelizard
Safer

Badge Winners

  • Integrate Spatial Data
    graysonhas earned the badge Integrate Spatial Data
  • Student (Grey)
    lenabaghhas earned the badge Student (Grey)
  • Welcome Back
    lenabaghhas earned the badge Welcome Back
  • Welcome Back
    dcoxhas earned the badge Welcome Back
  • Welcome Back
    mglimsdalhas earned the badge Welcome Back
Show all badges

Community Stats

31,808
Posts
120,815
Replies
39,431
Members

Latest FME

Terms & ConditionsCookie settings
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.