- - Community Resources
Ideas
Product Updates
Events
Blog

Home
Forums
FME Form
Data

Data

Using readers and writers to extract and load data.

6,700 Posts
24,702 Replies

When you subscribe we will email you when there is a new topic in this category

6700 Posts

Recently active

Most replies Most views Newest first

fmelizardContributor

asked in Data

Database listeners to send real-time e-mail notification with FME Server

Hello,I want to create a workflow that will monitor updates to tabular SQL Servers tables, which would send email notifications when my workbench has updated "error" tables. My workspace is effectively processing data and writing these error tables. But I am unsure of the best way to create database "listeners" that would note when new data has been added to an error table and then send out an email notification via FME Server. We have FME Server up and running and it does send email notification for when the workspace fails to run. But we have never sent email notifications for when a SQL Server table has been updated. Can anyone provide some suggestions on how to best do this with some links and possibly an example work bench that "Listens" for database updates? Thank you!

6 years ago

shweta

asked in Data

I want to saperate out jpg files from a folder into another folder querying based on character present in jpg file names. For example of any file has Character "A" then those files should be copied to my output folder.

Details :I tried to add Dataset & Pathname reader followed by StringSearcher, FeatureReader and writer as format JPEG.I am getting desired output till feature reader , when i checked with logged but i am not getting files in output folder.

6 years ago

reinaard

asked in Data

MapText Labeler text offset

MapText Labeler seems to have an issue over offsetting lower case text characters g,j,p,q and y when the output is dwgThe source data is a shp fileI use MapText Styler to create a text buffer around each line.In MapText Labeler I set the Line Rule Properties toHas anyone experienced this before?RegardsRazza

6 years ago

jim

asked in Data

Updating field value in ArcSDE table

I am trying to compare and update a field in a table. I have several tables from sde that require a date field to be updated. This worked in an earlier version of FME. In the workspace I brought the tables in and duplicated on writer. In the latest version of FME I get the following message:"Using ArcSDE Writer to write Spatial Database Engine 3.0 dataset `gis'ArcSDE Writer: Connected to version 'SDE.DEFAULT'Writer `SDE30_2' of type `SDE30' using group definition keyword `SDE30_2_DEF'Required columns 'GlobalID, ' not specified for predefined table 'SDE.SWGRAVITYMAIN'. Using Definition from DatabaseGrid size 0 (-6) for table 'SDE.SWGRAVITYMAIN' doesn't match that specified in mapping file (0)Existing SDE table 'SDE.SWGRAVITYMAIN' allows the following shapes: SDE_NIL SDE_LINE SDE_SIMPLE_LINE SDE_MULTIPART . Shapes specified in mapping file for table allows: SDE_NIL SDE_POINT SDE_LINE SDE_SIMPLE_LINE SDE_AREA SDE_MULTIP

6 years ago

annachukicheva

asked in Data

Translation was Successful, but Process still runs...

Hello, i Need to write the data to ESRI Geodatabase(Personal DB) mdb: after start the process -> the data will be writen, message 'Translation was Successful' will be written as well, but the process runs until the button 'stop translation' will be pushed. If the Writer is disable, and put inspector instead: everything works fine. Any idea? What could it be? Even if i get the data written, im afraid, that something goes wrong and will be done not until the end (as long as the process could not stop). p.s. i dont have any workspacerunner in workbench. Only Attributecreator and tester. I'm working with FME Workbench 2015.

6 years ago

david_prosack88

asked in Data

Building Spaces from walls

Hi All,I'm having trouble with developing room polygons from walls for a building from an autocad file. The difficult part is snapping lines together. I have a workflow, but it only completes about 50% and the rest will not budge. Attached is the workbench along with the CAD file. Any suggestions would be appreciated!David building-spaces.fmw

6 years ago

alexvsn

asked in Data

Bridge namespace prefix

According to the CityGML v2.0 standard, the recommended namespace prefix in case of Bridges class is "brid" When i'm using CityGML writer to output my model of Bridges, the inline definition of my CityGML profile contains the namespace and path to the .xsd schema file. In case of bridge the namespace writes :bridge", whereas in the case of other schema definition it contains the namespace prefix (tun for tunnel,bldg for building). Could this be a problem with the GML writer?

6 years ago

cesarp

asked in Data

MS_ACCESS_JDBC memory issue.

I can only use JDBC for my msaccess database in the office (no ODBC or MSACCESS DRIVERS in locked machine).. trying to load ~600,000 features causes the error : java.lang.StackOverflowError eventually.. tried changing as described here: https://docs.safe.com/fme/html/FME_Desktop_Documentation/FME_ReadersWriters/msaccess_jdbc/msaccess_jdbc.htm FME_JVM_MAX_HEAP_SIZE variable, re-started FME, rebooted machine, doesn't seem to do any different.. I know I'm close.. I just don't know how to make this variable setting to add more memory to Java..

6 years ago

pvaziri

asked in Data

how to speed up postgis reader

Hi,I need to read approx. 1.4M records within a single workbench process. No getting around it. They must all be read since they are participating in a FeatureMerger later and any one of the features might be a candidate.The reader starts out fine reading 2500 records a time under 10 seconds a pop. However this slowly degrades so that after 800K records it is now 30 sec. a pop and near the end close to 50. Soup to nuts, the entire read takes about 4 hrs and 45 min. RAM/Network speed really isn't a problem. So the gradual increase of record processing time seems odd.Is there any tip/trick for reading a large number of records (>300K ) from PostGIS (or any) reader so that the amount of time is significantly reduced? I was thinking of amping up the "Number of Records To Fetch At A Time" but I don't want to make matters worse.Thanks very much,Pete

6 years ago

andeh

asked in Data

Creating a postgresql function in fme - ERROR: unterminated dollar-quoted string at or near "$BODY$"

Dear allI am trying to compose a PostgreSQL function in FME using the SQLExecutor. My function installs correctly within pg admin, but since there is a dollar($) symbol included in the query, my function won't install correctly via FME and I receive the following error:Error executing SQL command ('CREATE OR REPLACE FUNCTION gv.create_tables_from_foreign_schema(_foreign_schema character varying, _local_schema character varying) RETURNS void AS $BODY$'): 'ERROR: unterminated dollar-quoted string at or near "$BODY$" The function looks like this:CREATE OR REPLACE FUNCTION $(p4_p3_schema).create_tables_from_foreign_schema(_foreign_schema character varying, _local_schema character varying) RETURNS void AS $BODY$;declare selectrow record;beginfor selectrow inselect 'CREATE TABLE ' || quote_ident(_local_schema) || '.' ||quote_ident(t.table_name) || ' AS SELECT * FROM ' || quote_ident(_foreign_schema) || '.' ||quote_ident(t.table_name) || '' AS qry from ( SELECT table_name FROM

6 years ago

carmijoContributor

asked in Data

Error writing to Excel - Feature is in writer mode 'UPDATE' but has no value for row ID attribute ''

I'm getting the following error when writing to Excel from a specific workspace using FME 2016.1.0.1.ERROR |Excel Writer: Feature is in writer mode 'UPDATE' but has no value for row ID attribute ''Despite what the error reads, my excel writer is set to insert. I'm also getting the error when saving the data from inspector to Excel. The problem seems to be specific to this workspace as i can write to excel without issue from others, with the same Excel writer parameters. I've tried writing to a different location without luck as well. Any ideas? Thanks!

6 years ago

abhaga

asked in Data

Convert nested complex GeoJSON to SQL Table

Hi All, I am trying to convert nested geoJSON to SQL table but FME converter is missing out on few tags when converting to a sql table (). The geoJSON file is attached below for reference. the FME SQL writer is not able to read "geometry" tag at line 75 from geoJSON file (). Thanks

Given 3 KML files, how could I merge them into a single KML while keeping them in separate folders ?

The source 3 KML's are similar in structure. The desired target KML would have 3 folders as shown below. Any advice or examples how to do it in FME would be greatly appreciated.

I've got output from 3 statscalculators coming together to create the following and was wondering how I could 'flatten' it such that that one record exists for each community and <missing> values have a value of zero.So the output would look like the excel image.

6 years ago

takashiContributor

asked in Data

How to ask a question regarding XML

How are you doing?I would gladly post answers to XML questions if I could, but some users tend to attach a full or invalid XML document regarding their question, so, honestly, I'm tired from parsing the full/invalid XML to find out the parts which are necessary to understand the question correctly.Could you please add a notification here about how to ask a question regarding XML? If there were such a notification, I could prompt that in my comment or answer. e.g.How to get a quick answer to your question on XMLIf you want to know how to extract specific values or parts from XML, how to modify some parts of XML, or how to filter features based on values within XML, it's recommended that you post XML fragments which are simplified to understand your requirement quickly.Extract your interest XML fragment from the source XML document with the XMLFragmenter.Convert the XML fragment to a Pretty-Print (human readable) one with the XMLFormatter.Using a text editor such as Notepad++, si

Hi guys,How would you retrieve attributes of raster KMZ file.In Google Earth, you see the actual attributes of raster kmz file, but in FME when reading it, it does not show them.Any ideas appreciated.ThanksDavid

6 years ago

surbhiContributor

asked in Data

ArcGISOnline Router time units?

I wanted to confirm that the unit of measuring time while using ArcGIS Online Router (Total_TravelTime in Routes) is minutes.

This should be simple. I am using "transformer" to change KML files to shapefiles. My GPS golf course sprayer uses the shp file to determine where to spray and where not to. It works fine and allows the sprayer to read the Shp file and only spray within the polygon. How do I make a similar shp file that only allows spraying to be done outside of the boundary? The shp file is in 2D. Here is an example. One day I may wish to spray only the fairways. The next day I may wish to spray the rough outside of the fairways and not have any product sprayed on the Fairway. Hopefully there is a golfer, ERSI expert out there who can make sense of this question

Hi, I have to convert a .csv file to o .Las file.

6 years ago

mcgregrrContributor

asked in Data

Is there a way to publish a feature class service to ArcGIS server without having ArcGIS Portal set up?

6 years ago

woltersContributor

asked in Data

PointOnRasterValueExtractor vs Point Shapefile

I want to know the value of a raster at specific points using the Point on RasterValueExtractor. Inputs are a raster and a point geometry shapefile (generated in another FME workspace).I simply connect the raster and point shapefile in the PointOnRasterValueExtractor (they have the same coordinate system), where they should be connected. Yet it gives the following error:PointOnRasterValueExtractor_2(VectorOnRasterOverlayFactory): Invalid point feature or feature with non-point geometry encountered. Only vector features with point geometry are currently supportedAre point shapefiles not supported? Do I need to transform my point shapefile in some way before using it for the RasterValueExtractor?

6 years ago

guy_lafleur

asked in Data

SDE30 Writer Crashes FME 2017

Currently in the process of trying to upgrade from FME Server 2015 / ArcGIS 10.2 to FME Server 2017/ArcGIS 10.5. We have around 400 scheduled replications that mostly write to an Oracle SDE database using the SDE30 writer. About 150 of these FMW's crash when run on the new environment. The log writes some warning messages (that I do not believe are related to the crash as they appear in the 2015 replications as well) and then silently crash. I have tried running with FME workbench 2017, and this process does not provide any additional logging.The warning messages before the crash are:FME Configuration: Using FME Reprojection EngineCreating writer for format: Esri Legacy ArcSDEFME Configuration: Destination coordinate system set to `BCALB-83'Coordinate System `BCALB-83' parameters: CS_NAME=`BCALB-83' DESC_NM=`Albers Equal Area, Province of British Columbia, Canada' DT_NAME=`NAD83' GROUP=`CANADA' MAP_SCL=`1' ORG_LAT=`45' ORG_LNG=`-126' PARM1=

6 years ago

steveatsafeSafer

asked in Data

What JDBC connections have you used in FME?

I recently updated Robyn's "Using the FME JDBC Format" article, adding a few more formats and a new category for the JDBC formats that FME now supports more completely, including documentation.It got me thinking... what have others attempted to do with the JDBC Reader/Writer? If you have any experiences - I'd be curious to hear about them and I could update the article.

6 years ago

svenkbh

asked in Data

How do I achieve truncacte and write of all features in one single transaction (PostgreSQL)?

I want to copy data from a source into a PostgreSQL table and I want the content of the target PG-table to change only if the whole operation is successful. My approach is to get the truncate and all inserts done in one single transaction. One way to do this is setting “Features Per Transaction” to >= number of features in the source (some hardcoded value, perhaps 9999999999999999999999). But is there a better way? In the documentation I’ve found info on setting “Features Per Transaction” to VARIABLE in combination with using a fme_db_transaction attribute. I’ve tried this, both setting fme_db_transaction to COMMIT_AFTER and to IGNORE. But the log file still says that there are 1000 features per commit (=the default number).

6 years ago

Page 163 / 268

Badge winners

ezrajudehas earned the badge FME Form Advanced Training
mosalahhas earned the badge Welcome Back
md4038601has earned the badge Teacher (Grey)
favoritesdishas earned the badge English Speaker
paulmcm601has earned the badge Teacher (Grey)

Show all badges

29,719: Posts

Terms & Conditions Cookie settings

Sign up

Already have an account? Login

An FME Account is required to contribute

Login to the community

An FME Account is required to contribute

Enter your E-mail address. We'll send you an e-mail with instructions to reset your password.

Enter your e-mail address

Back to overview

Scanning file for viruses.

Sorry, we're still checking this file's contents to make sure it's safe to download. Please try again in a few minutes.

This file cannot be downloaded

Sorry, our virus scanner detected that this file isn't safe to download.