Skip to main content
  • EN

Data

Using readers and writers to extract and load data.

  • 7,082 Posts
  • 26,079 Replies
7082 Posts
clang
Contributor
clangContributor
asked in Data
Generate Postgres backup files

Hi Guys,I have an ETL which writes to a Postgres database, before the writer begins the translation I would like the writer to run a script in Postgres to backup the database schema (including tables and indexes) as either a backup or sql file. I can run a script to backup/dump the entire database however I would just like to backup the schema.Creating a backup of the schema using the pg dump utility within PgAdmin 4 looks something like this--file "P:\\\\FINANCEBBK.BAC" --host "myserver" --port "5432" --username "postgres" --no-password --verbose --role "postgres" --format=c --blobs --encoding "UTF8" --schema "finance" "mydatabase"Any help greatly appreciated,Chris

15210
virtualcitymatt
Celebrity
mr_fme
Enthusiast
mr_fmeEnthusiast
asked in Data
Write DWG in Postgis

How to write data in Postgis with the same color as DWG, after recording in postgis the recorded data comes out with the same color.

750
mr_fme
Enthusiast
A
aquamarine
asked in Data
Contour labels with Labeller / LabelPointReplacer

Hi, I was wondering how I can remove the point that is generated with the text in the Labeller and/or LabelPointReplacer transformers? I want to generate contour labels but I don't want the additional point.I am using FME2018.0 Desktop.Thanks,

530
A
F
florian_at_ids
asked in Data
While reading WFS data got a XML error

Hi all,This is the first time I try to read WFS datas and I got some trouble.When I got to the parameters, eveything appears to be fine :Then when I tried to read it , this happened :The whole error text is : ERROR |XML Parser error: 'Error in input dataset:'http://wxs.ign.fr:80/---cryptedkey?---/geoportail/wfs/?SERVICE=WFS&VERSION;=2.0.0&REQUEST;=GetFeature&TYPENAMES;=ADRESSE-PREMIUM.LIEN.HEXA%3Alien_adresse_hexa&NAMESPACES;=xmlns%28ADRESSE-PREMIUM.LIEN.HEXA%2Chttp%3A%2F%2Fwxs.ign.fr%2Fdatastore%2FADRESSE-PREMIUM.LIEN.HEXA%29&OUTPUTFORMAT;=application%2Fgml%2Bxml%3B%20version%3D3.2&COUNT;=30000' line:6 column:3 message:expected end of tag 'hr'' ERROR |The XML Module halted on error, see the logfile for detailsERROR |A fatal error has occurred. Check the logfile above for details I guess that there is a problem in the XML, lacking something in it. When I ask the data provider, they say that can open it with no troubles ... so here I am, a

1860
F
M
mattyryan
asked in Data
Connect to FME Web Socket Subscriber with Pure Javascript

I'm attempting to use the standard javascript web socket to subscribe to the FME server web sockets endpoint.I'm simply doing: this.socketConnection = new WebSocket(webSocketUrl);That^ code connects to the web socket on FME server but never receives or sends messages. Am I meant to include a protocol? Does anyone know why messages would not be coming through or not be sent?

151
M
M
mattjelbert
asked in Data
Is there a way to make updates in ArcMap sync with AutoCAD (or at least make that update process quicker?)

Hi: I'm new to this forum and just found out about FME. I was curious if there was a way to use FME to create a workflow that allows you to update changes to a .mxd in ArcMap then have the changes update to a .dwg in AutoCAD. Or at the very least make the process of transferring over pt/line shapefiles easier to AutoCAD for .dwgs that need regular update maintenance. In particular I'm thinking of a city/municipality utility map where changes are made on the ArcMap side then translated to the AutoCAD side (or vice versa as many engineering firms tend to create construction/as-built drawings with AutoCAD). Any thoughts?

70
H
W
westdakota
asked in Data
Change Detection with LiDAR

I'm hoping someone can point me to a step-by-step article or post on how to use LiDAR from 2 different years to detect changes in topography. I've found this Q&A; Forum answer...https://knowledge.safe.com/questions/2487/lidar-feature-extraction.html... but it's sort of a broad-strokes overview of the process, and not detailed enough for me to do anything with, having never tried this before (on any platform).Thanks for any help you can offer!P.S. - My organization is using FME 2016.

410
H
dmerrick
Contributor
dmerrickContributor
asked in Data
FeaureWriter FileCopy Fails

Hi,I have a workspace with an Reader (SDE Connection) writing to multiple different zipped folders before a FileCopy Writer inserts a common disclaimer into each zipped folder. I want to use the FileCopy FeatureWriter due to the output port would then allow me to connect to a S3Uploader transformer and simplify my workspace down from 2 to 1.I however receive a error of: Cannot have multiple factories with the same name. Multiple factories were assigned the name 'filecopy_FeatureWriter-FileCopy_d4291b7d_cbed_4650_990f_3e25feda4a160'Am I missing some parameter setting or is there some workaround using the FileCopy Reader that allows processing after the disclaimer insertion?

330
dmerrick
Contributor
E
eugine_ta
asked in Data
Quality settings for RasterDEMGenerator

 Hi all,we are company that interested in using FME products. Main issue that we are blocked in current process is DEM generation. We compare results with Civil 3D and failed to tune FME workbench to get comparable quality. As I understood FME has only 2 settings for DEM interpolation: planar and constant. Planar gives us better results, but still interpolation doesn't seem good enough. It is not accurate. Ive attache some synthetic file that we use for quality assurance. And results from FME and Autocad Civil3D. Another question is what should be best input format for such operation, our pipeline set to work with 3d edges, but may be TIN would give better quality? May be anyone solved this task in the past to move process of DEM conversion of TIN from Civil3D to FME?Here is file that I am working with and results: files . Basically I dont understand why FME give so big difference in elevations and why it is not symmetrical for pyramids.

110
nathanatsafe
Safer
S
saqibamin
asked in Data
comparison of data

I have two files a csv file and an Excel file with 4 worksheets, I need to compare the two files to see if all records in the Excel file with all sheets are present in the csv file, there is a particular column in the Excel file, it is a string e.g. 'ff_abcd_12345', I need to get everything apart from the numbers from this and compare it to a field in the other file. There are thousands of records which need to be compared, all I need to do is check if the record exists and report it as matched or unmatched. Any ideas on how I deal with this will be much appreciated.

2368
takashi
Celebrity
C
cbrown
asked in Data
Grouping items on PDF pages

I have a workbench which writes individual objects to multi page PDF. The page number is defined using 'count' and assigning a 'group' which is then used after the PDF PageFormatter to specify 'pdf_page_number. However I now need to have 2 objects per page. Can anyone advise how I can group these automatically into groups of 2 to write per page?

720
C
redgeographics
Celebrity
redgeographicsCelebrity
asked in Data
Connecting to Bentley ProjectWise on-premise

I'm working with a client who has Bentley ProjectWise on-premise. We're trying to connect to that using the ProjectWiseConnector but we're having trouble figuring out the base url. pw:\\\\[servername]:[Repository] seems to work on the browser, but not in FME.Has anybody else done this before?

340
redgeographics
Celebrity
D
dmingl
asked in Data
Socrata Writer: Failed to Execute

When trying to upsert a large dataset I get the following error: Socrata Writer: Failed to Execute `""C:\\Program Files\\FME\\jre\\bin\\java" -Xmx1g -jar "C:\\Program Files\\FME\\plugins\\socrata\\datasync.jar" -ph true -sc "C:\\Users\\dmingl\\AppData\\Local\\Temp\\FME_1533665505412_4848\\control.json" -i u8yq-kcrr -c "C:\\Users\\dmingl\\AppData\\Local\\Temp\\FME_1533665505412_4848\\config.json" --userAgent "FME/2016.7.23.16717" -f I'm using DataSync 1.7

140
fmelizard
Safer
M
mp7
asked in Data
XLSX to DWG with template containing specific block

Hello,I'm totally new to FME and learning from scratch and I've already did some coordinate system transformations (fomr xls to xls and dwg to dwg), but now I'm trying to transform a list of point from an excel file(name, x coordinate, y coordinate) into a dwg with a dwg template for the point name.Now the points creation is easy but the problem is with point name that should be as a "block reference", just like the one in the template.I'm not even sure if it DWGStyler is the right choice.Can anyone help me out?Thx.templateworkflow

1310
M
J
jonathan
asked in Data
Generic writer - set at runtime

Hi All, Does anyone know how to set the FORMAT for a Generic Writer at run-time? I have a generic reader and want to output to whatever format the input data was. Is there a good way to do this?  I've tried getting the input fme_format_short_name and using that as an attribute, as well as OUT_FORMAT, OUTPUT, FORMAT, and OUTPUT_FORMAT. But all fail with the tautological:  ERROR |Generic Writer: Cannot use the Generic writer to create a Generic writer. Please select a different writer for the FORMAT keyword.  Thoughts? Thanks, Jonathan

260
jdh
Contributor
M
martyn27
asked in Data
Cursor error when mass reading from an Oracle database

Hi all, fairly new to FME and can't find much about this topic. I am using an FME workbench to move over a lot of data from an oracle database to a Microsoft SQL server. This is done by using a lot of SQL creators. It has been working fine but I think I have reached the limit of the Oracle I am using. My error is "did not succeed; error was `ORA-01000: maximum open cursors exceeded". I was trying to keep this all in one workbook - but I am not sure how with this error? I tried creating a separate database instance but that didn't work.

2060
erik_jan
Contributor
A
ac_ac
asked in Data
Prevent MongoReader from skipping attribute

As beginner with FME, I try to read documents from MongoDB.As far I understand, as a defaut behavior, if an attribute is not detected in the first read documents, its reading will be skipped. Is there any way to avoid this behavior?Many thanks!

42
A
R
rohan
asked in Data
WFS service fails to write to featurewriter

I am attempting to write from a WFS FeaturerReader directly to a FeatureWriter, using the tip outlined in this post:https://knowledge.safe.com/questions/69654/writing-table-with-featurereader-featurewriter-pro.htmlThis works for some datasets, but when I try to use a particular dataset, I get an error message:Bulk copy failed on table 'public.road_works_line_temp' using delimiter ':'. Error was 'ERROR:  extra data after last expected column CONTEXT:  COPY road_works_line_temp, line 1: "NULL:featureMembers:rwe_line.143236:143236:Sporting/social event:NULL: road closed:NULL:Pending:NULL..."'The wfs url is http://api.vicroads.vic.gov.au/vicroads/wfs?SERVICE=WFS&VERSION;=1.1.0&REQUEST;=GetCapabilities&AUTH;=eyJraWQiOiJRRFFQWDZVSDlQRExOOU9GQVowMlNFRFVYIiwic3R0IjoiYWNjZXNzIiwiYWxnIjoiSFMyNTYifQ.eyJqdGkiOiJuUzRMM0MzSURvdzRsakNNc3R2Y0IiLCJpYXQiOjE0OTE1Mzg2NDQsImlzcyI6Imh0dHBzOi8vYXBpLnN0b3JtcGF0aC5jb20vdjEvYXBwbGljYXRpb25zLzRBeTd5RjJtUUNpQlpxUHo5Q3lRVTgiLCJzdWIiOiJodHRwczovL2FwaS5z

450
R
F
frtou
asked in Data
GeoJson to ESRI GDB - Manage field Alias

Hello,I convert data in Geojson format to ESRI Geodatabase. I would like to add alias to each field. How to do ?

180
siennaatsafe
Safer
R
rclarknxf
asked in Data
Help with Web Socket Subscription

I've followed https://docs.safe.com/fme/html/FME_Server_Documentation/Content/AdminGuide/configuring_for_https.htm#EnableSSLWebSocketServerfor HTTPS, service URLs changed etc. When trying to create a websocket subscription under Notifications I get an error of "There was a problem communicating with the REST API" when I click validateFME seems to be running fine and I can query the REST API ok.Any ideas?

80
richardatsafe
Safer
K
kbachmatiuk
asked in Data
DXF export with fill color for areas

Hello,I have a problem with exporting dxf file. I use shp file reader to import polygons and i would like to export them to dxf format (dxf file will be used later in old version of GlobalMapper). I dont know how to export dxf with fill colors. Which writer should i use and which options must be selected to get polygons (not lines ) with fill color.

2360
K
bubblebeb
Contributor
bubblebebContributor
asked in Data
Google Earth Writer - kml id doc

I am writing some geom to a KML file and wanting to group the data by Phasing. I have used the feature type name to identify the grouping. When it writes the KML the groups are shows as kml_id_doc_1.1.11.4kml_id_doc_1.1.11.5kml_id_doc_1.1.11.6Does anyone know what is going wrong and how I can fix it?Thanks

50
jovitaatsafe
Safer
T
tsurveyor
asked in Data
processing dgn error f_67(CreationFactory)

My workflow successfully read in the 600,000 dgn features, ran through a Matcher transformer to eliminate duplicates, was going through a featuremerger and geometry filter where it bombed as it was writing to shapefiles. The error - f_67(CreationFactory): Error - Number of measures found on the 'line/path' geometry did not match features number of coordinatesI have my Reader set to Master coordinate units. Has anyone seen this error and is there either a setting or some other filter/transformer that can be added to fix this?I'm reading MicroStation V8, processing in FME 2017

60
T
matself
Contributor
matselfContributor
asked in Data
How to read folders and pipe filenames for processing?

Hi!Working with Sentinel data, you get some 120 files i a zip file (standard download). Only three are needed to make an rgb image with FME (as described in this post: https://knowledge.safe.com/articles/43742/making-rgb-images-with-sentinel-data.html).This workspace snippet begins to extract these file names and should then pipe them to the next step (there is more to follow). But I am unable to figure out what goes into the gap between the TestFilter and the RasterBand... transformer. I simply need to get the path_windows attribute to be understood as an input file.

4570
takashi
Celebrity
D
david_prosack88
asked in Data
Online IMDF Validation

Hello y'all,I submitted a large international airport file to your online IMDF validation portal and I haven't received an error report yet, it has been over 45 mins since I submitted the file. I have uploaded a smaller file, and I received an error report. Is there a file size limit with the online validation? I appreciate any guidance! I am using FME Workbench 2018.1.Thank you!

160
D

Badge Winners

Show all badges

Community Stats

31,790
Posts
120,760
Replies
39,405
Members

Latest FME

Terms & ConditionsCookie settings
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.