- - Community Resources
Ideas
Product Updates
Events
Blog

EN

Home
Forums
FME Form
Data

Data

Using readers and writers to extract and load data.

6,449 Posts
23,856 Replies

When you subscribe we will email you when there is a new topic in this category

6449 Posts

Recently active

Most replies Most views Newest first

alisonsepContributor

AGOL writer-is there a limit?

Hi, I have one Form workspace that writes to a hosted feature service on AGOL fine. It is set at insert and Truncate. It writes a little more than 9,000 features and I run it every week without issue. However, I now have a different hosted feature service on AGOL that has +23,000 features. When I try the same process to update it, it drops all the data and only ends up writing out 15,000 records at most. The number of records diffs as well. I have tried adjusting the timeout and features per request, but have not been successful in achieving the full write out. Any suggestions would be appreciated.

fmesafe.podporaContributor

Sequence of points

Hey. I need some advice, or at least direction, on how to solve a complex problem. See attachment: I have a track. The track consists of waypoints, the waypoints are minisegments which consist of a startpoint and an endpoint and these minisegments form one big segment and these big segments form one complete track. The direction of the "flow" of the points is determined by the number of startpoints and endpoints, see below, where we see that the only possible direction is from A to E, since it must connect startpoint to endpoint between the minisegments, which form one segment out of three. The waypoints of one of the segments at the level of the points that make up the minisegments look like this - A, F, C, E: Startpoint: AEndpoint: FStartpoint: FEndpoint: CStartpoint: CEndpoint? EThe problem I'm solving is that some segments have this sequence of startpoint and endpoint reversed in the source data, so that the sequence of points of one of the segments would look like this: E, C, F, A

jinseokContributor

Issue of visualization failure after data conversion (obj to citygml /obj)

We have tried to export(write) an obj 3D building file to dae / citygml / slpk file formats, and all three of the conversions were successful.However, only the dae format was successfully visualized in a viewer, and citygml/slpk failed to visualize with the viewer that supports the formats.Could you give some advice for this issue?CASE1 - obj to citygml (failed to visualize)input 1784_OBJ.zip output 1784_CityGML.zip output import view capture (KITModelViewer) No geometry information, only attribute information can be verified CASE2 - obj to slpk (failed to visualize)input 1784_OBJ.zip output 1784_slpk.zip FME Capture output import view capture (ArcGIS Pro)

Write an IFC in Postgres with Postgis extension

Hello, I have an IFC file I wish to import in Postgres(PostGIS extension) to go through spatial analysis. Oriinally, what I did was to convert the IFC to a 3DGIS compatible format (GML) and import it in Postgreswhich was successful. Exploring a little more, I saw that Postgres supports the solid geometry type of FME and I performed the same conversion, without converting the geometries to b-rep. This photo shows the input in Postgres and this photo shows the output when I am reading it from Postgres. The highlighted part is visualised differently. I couldn't help but wonder the reason why.Could you please help me figuring this out?Thank you, George

Data Inspector 2023 can't read ArcGIS Pro FileGDB

I’m trying to use the Data Inspector to inspect data in a Pro-generated file geodatabase. I’d get these errors which prevents any dataset from coming in.I’m on ArcGIS Pro 3.1.3. Dialog window:“Failed to open datasetEsri Geodatabase (File Geodb) dataset ########### could not be opened as configured. See log window for details. Remove it from recent dataset list?“Log window:An error occurred while attempting to open the table, feature class or relationship class 'Administrative_Lines'. The error number from ArcObjects is: '-2147211773'. The error message from ArcObjects is: {The item does not have a definition.}It could have failed to open because it contains ArcGIS Pro-only features, for example Arcade expressions. If this is the case, please install ArcGIS Pro or Pro-compatible ArcGIS Server and 64-bit FME. If already installed, set the Esri ArcGIS Compatibility setting for this workspace to 'Esri ArcGIS Pro' under Navigator -> Workspace Parameters -> Translation From FME Form, I

jtermaatenContributor

How to use Deployment Parameters in a FeatureReader to reference a Custom Resource with a local value set to a Share?

We're using FME Flow 2023.1.2.We have got a Windows Share registered as a custom resource at FME Flow, which is locally accessible as a UNC path. I'm looking for a way to reference the custom resource in such a way that it locally uses the UNC path and at FME Flow automatically uses the resource. I believe I can use Deployment Parameters for that.When I create a Deployment Parameter and use it as part of the path in a FeatureReader, it doesn't recognize the file I'm trying to reach, though I did fill in the local value of the Deployment Parameter.Deployment Parameter Value: $(FMEShare) Deployment Parameter Local Value: \\computername\shared_folder\ Reader Path: $(FMEShare)Subfolder\file.xlsxI tried to create an attribute _pathExcel that is set to the path I tried to fill in in the FeatureReader. Locally that worked, but at FME Flow I got this error message:Too many recursive macro substitutions -- line currently is: `$(FMEShare)'How can I tailor this method to get it to work? Thanks!

topotomaContributor

Transpose excel data by feature ID

Hello Community,I have a question and looked for answers but didn’t find it yet.It must be simple for someone but I have not yet figured it out. I tried attribute transposer, aggregator etcI have a table with a feature which has an ID here shown as Feature 1,2,3 and multiple columns with values for each month of the year.It is like so: What I yould like to get is the following:In the end I need a row for each column corresponding to the feature it is refering to.I found some sort of a possibility but it it very “manual” an inelegant in FME as I have to create each month manually and associate a date etc. If someone has a better idea, I am very interested about it. It would help me in the long run as I may have a wrong approach to transposing data with FME.Many thanks to all the contributors. Best regards, Thomas

hiyhelgContributor

attributeValueMapper

Dear FME community,I hoop you will help met with the problem I am try to solve . I am reading WFS data which has more than 200,000 records. I just want to replace one of the fields from my own Excel file. I used to Attributevaluemapper to replace the value of fieldname owner with the value name owner from my exce list. I am not succeeding Do you have any idee how to tackle this?Any help is apricated and thanx in advance!!

gael123Contributor

Distance to coastline

Hi All I have a point distance data set in a parquet file with x y coordinates and a geom column with geometry encoded in hex binary and i have line data for coastline I want to calculate distance between these points and the coast polyline .In my workflow am using geometryreplacer to get the geometry then projecting both layers using csmapProjector then using neighborfinder but am getting unexpected results ...is there a better approach

fbastian73Contributor

How to trace eletrical network to get desired output

Hello experts,I need suggestions or some help on getting FME workbench to output 2 excel files based on a user parameter which the user will be asked to supply before the workbench begins to run.Both files will be the result of tracing a feeder (the user parameter) downstream from its origin at the feeder/hypernode point.The first file I am trying to output should contain something similar to the below.So up to column G is trace and H and I is result of spatial intersection. This is based on the above screenshot which shows the feeder extension (brown line). The hypernode is located at the zonesub (blue square) and then we want all reclosers (red and white circles) along the feeder to be captured and outputted as well as the fire polygon(s) the feeder intercepts.The second file I am trying to output should contain the ids of any spans/linear objects which follow downstream of the point object until the line comes to another point object (recloser) and then carry on with that device and

hoa_leContributor

How can I make MapboxStyler for polygon file?

Hi,I have a polygon file. I want to make MapboxStyler for it. But I don’t see it styled.I use parameters as the below:Could you tell me why? Thank you

gennineContributor

How update existing table saved to PostGis with FME

Hi, I have acheived an attribute update in an existing table in QGIS.The transfert to PostGis works well when I use "Insert" and "Drop and Create" handling.However, I can not figure out the operation with "Update" and "Use existing" in order to properly transfert the updated value of the attribute code_dep. Any help is welcome.

lambertusEnthusiast

Splitting a polygon with another polygon

My goal is to split the green polygon at the edge of the pink polygon. However, the pink one is not aligned properly. The black line indicates the cutting line I want to achieve. I am thinking of creating a buffer around the pink dataset, however only at the top and bottom border. At the left side, where the cutting line is present, a buffer is not allowed. When such a buffer is created I could use an Intersector probably to split the green polygon. Any support is appreciated!

ed-progresContributor

How to sort column names in Table View (Visual Preview)

Hi FME folks,Is it possible to sort the column names in the Table View in the Visual Preview alphabetically?It's frustrating to have to scroll all the way to the right to find a column that starts with an “a”... Thanks!

simeonContributor

Expanding JSON output with an array into features

I’ve tried searching for answers to this but I’m sure I must be using the wrong search terms as I get close matches but nothing that has pointed me in quite the right direction yet. I’m hopefully that someone can help me understand how to expand the following JSON fragment into a set of features, where each feature contains the same initial set of fields, but only contains one of the final array entries.Here’s an edited example of the JSON I’m dealing with and how I’d like the out to look. {"UPRN":"12345678","CaseNo":"15/01234/ABCDE","Address":"16 Some Rd Surrey SM1 1AA","Name":"Miss T Mountain","DocumentID":["7654321","7654322","7654323"]}I’m trying to extract this into 3 features, each of which includes the uprn, CaseNo, address, name and a DocumentID. I’ve been able to extract the whole array of documentIDs but haven’t figured out how to extract each one in turn and pair it up with the remaining information.The JSON output I’m dealing with contains a number of such records, in each

galigisEnthusiast

Export Photos as JPG files through API Fulcrum

Hi All, I’m working on a workflow to export the photos (as .jpeg) collected with the Fulcrum app in the filed for a bridge inspection. I have been able to install the Fulcrum Web Connector from the FME hub here and make the relevant connection through a Token. Also I have put together the following workflow to read and extract the photos but for some reason when I paste the URL photo in my browser says ‘This page isn’t working’. I expected to view and download the photo through the API / URL (picture).Here is the workflow created so far: I think am not too far off to get this working but would need some advise if am doing something wrong as can’t view the jpg files? example of the URL here.Also below an example of one of the photos extracted in my C drive:Any idea / trick will be appreciated.

galigisEnthusiast

Extract JPG files through Fulcrum API

Hi All,I’m working on a workflow to extract JPG files using the existing Fulcrum API. I have been able to install the Fulcrum web connector from the FME Hub and make the relevant connection to my organisations using a Token. All good.The problem is am not be able to view the URL photos in my browser as shows ‘This page isn’t working’. Below the workflow to extract the photos:Creator HTTPCaller JSONFragmenter AttributeFileWriterBelow a screenshot of some of the URLs in the JSONFragmenter:Does anyone know why I am getting the This page isn’t working error?

sekarjntuContributor

Character encoding conversion

Dears, we have recently upgraded fme server windows machine to 2022. Post that character encoding default changed from Ansii to UTF8 because of this files generated by our workspace having special french characters causing issue for interface systems. Earlier with ansii encoding format they were able to read files with out issue. Im not able to find how can i change to Ansii encoding fo rthe files generated by our workspace. As of now im opening the file in notpad and saving in ansii and then sending to them as a work around. Pleas ehelp me ot automate this utf to ansii conversion. Thanks

koenvdwContributor

geotif writer always uses jpeg compression

Hi guy’s A workbench that worked before gives me some crazy errors that’s driving me nuts.I used the workbench before and it worked normally. It has to write some geotif tiles with deflate compression. No matter what I do, I always get errors about me using jpeg compression. I tried everything. Clearing cache, deleting cache manually, restarting pc, copying everything in new FME file, updating all the transformers, making a new wb, try running it on another pc with more recent version of FME... I had a similar question in the past and than the solution was to use a bulkattributeremover with ‘geotif_’. This doesn’t work either. This is the wb, it’s just shapefiles and geotifs as import: Any one with ideas?

plancet10Participant

Migrating Attachments GN to UN

I am currently trying to migrate my Geometric Network (GN) system data to the Utility Network (UN) and would like to include attachments in the FME workbench. Currently, I have a workbench that successfully migrates the points, lines and polygons (Assets, domains, and attributes) to target UN features but I am not sure how to handle attachments. Should attachments be handled in a separate bench?The attachments would be mapped from the source tables to the new UN attachment tables. For example, hydrant attachment table would be migrated to Device attachment tableCan attachments be migrated using FME? What type of transformers and workflow needs to be utilized to move this type of data?

arnovananrooijContributor

PostGIS polygon turns into Geopackage multisurface in dynamic mode

Hi all,I see some strange behaviour when converting a simple 2D box from PostGIS to Geopackage in a dynamic mode (using schema). I have attached a simple workspace which demonstrates the strange behaviour. A simple box is created in 2D and writen to PostGIS as a postgis_polygon. I ckecked the table in the database and is has the correct definition for the geometry (geometry(Polygon, 28992)). After the creation the table is read wit a FeatureReader of type PostGIS with the same connection parameters. It is set to read the schema and the features. The result is directly written to a geopackage with a FeatureWriter. Geometry is from schema definition. When i open the result in the Data Inspector i see the geometry type has become geopackage_multisurface where i would expect it to be of type geopackage_polygon. At the moment we are upgrading our FME environment (Form and Flow) from FME 2021.2.6 to 2023.1.2 and ran into this problem. In FME 2021.2.6 i did not see this behaviour. This parti

ravibansalContributor

Not able to see more than 100 records in visual preview table

In FME 2024, i have feature caching enabled but when I run the workspace, I cannot see more than 100 records at any transformer in the workspace. And if I try to sort them in asc or desc order by clicking on any column, nothing is displayed.

jfisch25Contributor

ArcGIS Pro Data Interop Write to ArcGIS Online Geodatabase won't delete

I’m using Pro 3.2 and Data Interop extension with a simple workspace to pull point features from a SQL server database and publish them as a layer in AGOL, all within AWS on an ec2 instance. I have this scheduled to run every 15 minutes so users get fresh data. The workspace also uses an HTTP call to make the layer editable just before overwriting it, and then a second one after writing, to disable editing, because the layer is public. Seemingly intermittently, the file geodatabase used to publish the layer will not get deleted. I have the fgdb that was originally used to publish the layer still there in the same folder as the layer, but every so often another fgdb will be created and left in that folder too, so I get duplicates. Some mornings I check and there are 3-5 or more of them. I can manually delete them, but I’m trying to determine why they’re not getting deleted in the first place. I’ve checked the logs and can’t see anything out of the ordinary. I tried adding a 30-second De

ravibansalContributor

How do I write Lat Long to a non-spatial table with XY coordinate?

I have a non-spatial table with about 5 million records of XY coordinates. I need to update the table with the Lat Long values in the so-named columns.I am facing problems with writing these to the table. When I choose UPDATE/UPSERT/fme_db_operation, the writing takes a seconf for every 10 records. It’s impossibly slow.When I choose, Insert with DROP and CREATE or TRUNCATE EXISTING, the script gets stuck with following last 4 lines in the log: Writer `MSSQL_ADO_2' of type `MSSQL_ADO' using group definition keyword `MSSQL_ADO_2_DEF'Microsoft SQL Server Non-Spatial Writer: Opening `GISReporting' for write operationMicrosoft SQL Server Non-Spatial Writer: Read 1 DEF line(s). Found 9 attribute(s)Microsoft SQL Server Non-Spatial Writer: Trying to connect using Microsoft OLE DB Driver 19 for SQL Server...MSSQL_ADO writer: Splitting bulk features into individual features NOTE: I have set BULK Read and Write at 100000 records for every transaction.

albjohContributor

Dynamic truncate writing to SDE

I’m having trouble writing dynamically to a SDE feature dataset using the ‘truncate’ option in FME 2023.1. The features are written, but there is no ‘truncating’ taking place before the writing and instead the data doubles up... It works as expected in FME 2021.2.6 however, maybe due to the different way in which the feature dataset is specified in this version under ‘Table Creation Parameters’. How do I make it truncate also in 2023.1 version? Current set up in writer for FME 2023.1:

1
2
3
4
5
6
...
258

Page 3 / 258

Badge winners

dbaldacchino1has earned the badge Student (Bronze)
dbaldacchino1has earned the badge Student (Grey)
dbaldacchino1has earned the badge Expert (Grey)
dbaldacchino1has earned the badge Teacher (Silver)
dbaldacchino1has earned the badge Teacher (Bronze)

Show all badges

28,772: Posts

Powered by Gainsight

Terms & Conditions Cookie settings

Sign up

Already have an account? Login

An FME Account is required to contribute

Login

Login to the community

An FME Account is required to contribute

Login

Enter your username or e-mail address. We'll send you an e-mail with instructions to reset your password.

Username or e-mail

Back to overview

Scanning file for viruses.

Sorry, we're still checking this file's contents to make sure it's safe to download. Please try again in a few minutes.

OK

This file cannot be downloaded

Sorry, our virus scanner detected that this file isn't safe to download.

OK

A product of

Company Products Giving Back Careers Contact Feedback

Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.