- - Community Resources
Ideas
Product Updates
Events
Blog

Home
Forums
FME Form
Data

Data

Using readers and writers to extract and load data.

6,779 Posts
24,927 Replies

When you subscribe we will email you when there is a new topic in this category

6779 Posts

Recently active

Most replies Most views Newest first

jstanger

asked in Data

Get a list of valid reader/writer formats in a workbench

Is there a way in a workbench to retrieve a list of all the valid formats that the version of FME currently running the workbench can use? A separate static file isn't really able to solve my issue. I know it is a strange request.Essentially, we are using FME Server and will be using a Generic Reader to parse all user provided data then filter by a pre-configured list of valid formats my workbench will read from. The clients require the ability to add new formats and I'm trying to see if we can create a workbench that validates their configuration file against what FME can actually do.

Issues with Duplicate Consecutive points in 3D in Features (when writing to ArcSDE)

I'm getting a ton of issues with 'Duplicate Consecutive points' when trying to write SHP to ArcSDE and the GeometryValidator is no helping in remedying these. About 25% of my polylines are being rejected. The Shapefiles come out of a commercial subsurface modeling software, so I have no control over how they are written.Is there a way to ignore the duplicate point? ESRI software isn't complaining about this issue with the data. But I sure prefer doing my work in FME.

Help with SurfaceBuilder

FolksI'm going slightly mad trying to get a transformation from shapefile to INSPIRE GML to run successfully. I've succeeded with about a dozen, but one particular file just refuses to play ball. When I first tried it, it crashed midway through, so I've spent a large amount of time trying to make sure the input data is free of errors. I imported it into PostGIS and ran validity checks, then ran it through the QGIS topology and geometry checkers (and corrected everything they threw up using some of the GRASS tools available, and manual editing where they couldn't correct the problem automatically), then connected the shapefile to a GeometryValidator transformer and it says there aren't any issues. But... when I try and run the whole process, it just hangs up. The sticking point seems to be the SurfaceBuilder transformer; when I connect an inspector to it I'm getting the following warnings:Dissolving a donut produced more than one outer shell. Picking a representa

132

4 years ago

mhabContributor

asked in Data

ArcGIS Portal Feature Service Connection

So I need to admit I am absolutely lost in an attempt to use a connection for an existing ArcGIS Portal Feature Service.I have spent hours to find my way to create a connection for the parameters I need in order to access an existing ArcGIS Portal Feature Service and I fail miserably. I get lost in dialogs where I feel like I need to learn Web development instead of just saving the infos I need in order to use it later on.The Information I have and I need to access the existing portal Feature Service is:UrlFeature Servicemethoduserpassword... and instead of allowing me to save a connection with these parameters I am sent on a journey through Feature service templates where I don't have the slightest idea what I need to input.This kind of frustrating adventure is absolutely normal for too many tools out there, but I am not used to this from my long time FME experience.Where have I lost my way ? And please don't come up with details like REST call so and so ;-( HeeelpMichael

Shapefile surfaces issue

I tried converting a shapefile to cesium 3dtiles but after conversion some wall surfaces of the shapefile buildings are missing.When i inspected the data in fme data inspector the missing wall surfaces are transparent.Why some surfaces are transparent, I have attached the image of the shapefile viewed in inspector .How to bring all the wall surfaces in the output tileset without missing.

Reader Dynamic CSV Schema Changes

I've been looking at all the various options people have for Dynamic Schema changes in a CSV reader. But the only thing that consistently functions seems to be the "Update Reader" Menu option.Scenario: I have a single CSV where the schema gets added to on a regular basisWeek 1: 'col1, col2, col3, 'week 1'Week 2: 'col1, col2, col3, 'week 1', 'week 2'Week 3: 'col1, col2, col3, 'week 1', 'week 2', 'week 3' etc.I need to be able to read in the CSV and output to a DBMS table, pulling in the new column each week.

It is creating a Multipoint feature class instead of point feature class.

Feature Writer : ESRI Geo Database(file Geodatabase)Even though I have selected GEODB_POINT as Geometry in feature writer parameters.Tried to convert in to point using vertex creator but no luck.Can anyone help me how to write point feature class instead of multipoint.

542

4 years ago

gogopotter90Contributor

asked in Data

Excel styler Column name 'A.formatting' for sheet 'cars' could not be used as a column position.

Excel Writer: Column name 'A.formatting' for sheet 'cars' could not be used as a column position.Error shows to me bcs i used Excelstyler before feature writer but iam using the option use attribute names as column position Yes and i need it . i know also it should A.formatting inside the user attribute but it occurs to terminate the program .so any suggestion ,how could i change back ground color of some cells in the first column without terminate the programThanks

4 years ago

jakethepainterContributor

asked in Data

Intermittent DB connection errors with FME Desktop 2019.2 32-bit

I had this problem beginning in FME 2018 32-bit and thought it was related to the JVM memory issue, but I upgraded to 2019.2 and I'm still having a problem connecting to oracle databases intermittently. Here's one case: I have an sde reader (non-legacy) that is set up using a path to an sde connection file. I go to import a second feature type to the same reader and I get an error:Unable to connect to Oracle database, because the database environment could not be initialized.If I manually type the featureclass that I want to add into the dialogue, it actually just loads up, but I can't browse the Table List to pick it. Another case is in the oracle non-spatial connection that I'm using with a sqlexecutor or sqlcreator. Sometimes I can create and test a sql string just fine; other times I get the same error:Unable to connect to Oracle database, because the database environment could not be initialized.If I completely close FME and reopen, sometimes it works again. Som

How to add sub layers to a main layer?

Hello, I have Roof, Wall, Floor, Building Part, Window and Door in FBX format. Once it is converted into citygml format the output i get is as follows The building layer contains the complete data set and the other layers contain the respective data sets as mentioned in the layer name. But if i turn off the building layer the data is not hidden as the other layers are independent and contain the different items. Attached below is the screenshot of my workbench. However I am looking for a different kind of output. The building should act as the main layer and other features should come in as sub layers for the main layer. Similar to the attachment below. This would let me turn off the building layer as a whole or I can turn off the different sub layer as per need. Please let me know if someone can help me to resolve this. Thanks in advance to everyone. Regards,

4 years ago

ninani

asked in Data

Read from dwg into excel sheet

Hi! i have to read dwg plans and export information from the title block to excel. Just few values are interesting to me. I have an excel sheet with possible attribute values so only values which are both in the Autocad and in the "reference excel sheet" go to the final writer in excel format.for example:Cad provides the attribute autocad_text_stringautocad_text_stringgreenThursdaywhite380012/01/2019FebruaryNAMEPeterExcel reference sheetColoursDays of the weekMonthsNamesblueMondayJanuaryLaurayellowTuesdayFebruaryMaryredWednesdayMarchAnnegreenThursdayAprilJuliaorangeFridayMayCharlesblackSaturdayJunePeter SundayJulyAnthony AugustBob September October November December The writer should look like this:ColoursDays of the weekMonthsNamesGreenThursdayFebruaryPeterSo the final table contains values that were in both places. The rest are not relevant. Does this make any sense? Hope to get help because I don´t know how to make the connection between my 2 readers in an effecti

392

4 years ago

simonhumeParticipant

asked in Data

add new column to .shp file at regular intervals

HiI am downloading a .csv file from a remote ARCGIS server, combining it with an existing .shp file and then writing out to another .shp file. This makes this one table for each instance of the remote .csv file being updated (which is daily). This is working fine.I am looking to produce another, master, .shp file which is updated every day with the data from the newest.csv file. Each release comprises 2 new columns in the master table.So this is the master table after the first release is insertedAnd this is the release after the second release is inserted The master .shp file will therefore have 2 new columns added on a regular basis.I am unsure how to insert the new columns into the master .shp file each day, ensuring the column names reflect the 2 additional entries. I'm thinking something like adding a counter to the column names I'd welcome any suggestions anyone could providethanksSimon Hume

4 years ago

stuartpaice

asked in Data

12DXML reader Pipe segments

Hi, I know this is still being developed but we are trialing the use of the reader to write GIS information. At this stage I am able to work around some of the export problems from 12d model BUT i cannot solve generating individual pipe geometry segments from a <string_drainage> object. any guidance would be appreciated. <string_drainage> <data_3d></data_3d> - Geometry <pit></pit> - Pit information (xyz) <pipe></pipe> Pipe information (no direct Geometry) </string_drainage>

How to create Attribute of Road Geometry

Anybody have idea about Geometry characteristics of road each 10 meters with Fields of Geometry table with output Shapefile1)Road2)Segment3)Length- in meters4)Shoulder- in decimeters5)Carriageway- in decimeters6)Additional lane- in decimeters7)Slow lane- in decimeters8)Platform- in decimeters9)Median- in meters10)No of lanes11)Slope- % of a pavement12)Camber- Cross slope (%) of platform13)Radius of curvature14)Geographical coordinates(lat & long)15)UTM coordinates(x,y,z)

155

4 years ago

jugoslaviaaSupporter

asked in Data

Is it possible to disable FFS file?

Hi all, I would like to ask whether it is possible to disable the generation of the FFS file. I am running into some storage problems. Therefore, I would like to disable the FFS if it is possible. Thanks in advance.

Reading from xml with one to many relationships and writing to spreadsheet

I am trying to read an xml file with a nested structure and output the results to a spreadsheet. I am using an identifier (effectively a primary key) and a number of attributes for which each identifier may have none, one or more than 1 value for the other attributes. Using list exploder I have managed to get the right output for the identifiers with multiple values, and the rejected output contains the values I want for the rest of the identifiers, but when I try to direct the output to a single spreadsheet I get only the indentifiers from the rejected values, and none of the rest of the attribute values. What am I doing wrong?

4 years ago

gvatin

asked in Data

Error on MongoDB3 connection test

While using the MongoDB3 Database connections to replace older MongoDB connection, I have this error log showing with the "Test..." button. This error also displays when a query is made on FME Desktop : the query works, features are correctly selected and used, but the error still exists without stopping the process.MONGODB3 reader: EstablishFeatureDefn(): not authorized on MY_DB_NAME to execute command { find: "system.views", filter: {}, batchSize: 2000, limit: 2000, $db: "MY_DB_NAME", $readPreference: { mode: "primaryPreferred" }, lsid: { id: UUID("a797ca3f-a6d9-4739-aa15-e6c2c5f71a6b") } }: generic server errorMy Mongo user is admin of the DB, I can't see why there is unauthorized action? Thanks for the info!I use FME 2019.2.1.0 (20191202 - Build 19813 - WIN64)Mongo version is 4.0.13

What are ffs files?Are these necessary to keep?

There are around 50 to 100 FME workspace translations running on my server. All these translations generate log files which I keep for 3 days. Along with log file FME also generates ffs files which consumes lot of space on server. Does it affect my performance time? One translation created 1 gb ffs file in temp folder creating space issues on server and affecting other translations.

Reading a segment of json that is geojson

I have a json where part of the json is in geojson.The "polygons" attribute is just pure geojson multipolygon.I'd like to able to read all the json structure but leave the polygons structure as a string and convert that to spatial data.I've tried jsonflattener and exploders + listconcatenator/featurereader with no luck.I know I can do something with python, but would rather not.I've attached a sample file if anyone would like to give it a try

AutoCAD 3D file adding extra lines in surface

When I bring in a 3D DWG file to make some bulk edits, FME is adding a lot of extra lines in the surface of the structures, triangulating vertices etc, these remain in a .dwg export. Is there away to clean this up so it presents correctly inside AutoCAD. Original FME DWG export Many Thanks

4 years ago

lotte_v

asked in Data

Image exceeds license ECW writer - FME 2020

I am trying to mosaic several ECW images together into one large file in the latest release of FME 2020. It is a very large file, according to FME it has a raster size of 1360000 row(s) and 1184000 column(s). And after the mosaic transformer has finished I get the following errors:ERROR - ECW writer: Could not open compression task Output image size exceeds license ERROR - ECW writer: Failed to write dataset 'G:\\test\\test.ecw'. This may occur if you have insufficient privileges to write this file, or if insufficient disk space is availableIs there indeed a size limit with certain licenses? I couldn't find anything about this. There is also no reason why it would complain about privilidges or disk space for the output folder, as there should be more than enough space available on the disk.How can I solve this error?

How to get the NAME of the FID field of a GeoPackage?

We made copies of our feature class tables and copied them over to several GeoPackages. Now I want to use the geopackages in a FME workspace to import data to the same table again.First step therefore is to validate the GPKG table to the DB table to make sure that the structure and fields are correct and dtermine the correct db table.To accomplish this, I use a dynamic reader and collect all field names (attribute{}.name), concatenae them and compare the resultin strings with the concatenated string of the db table.This generally works as expected but unfortunately the FID column is missing in the attribute{}.name list from the GPKG reader. I know that there is the geopackage_fid attribute but that attribute only holds the value and not the name of the FID column.is there a way to get the name of the fid field of a GPKG?

Azure Synapse the new SQL Data Warehouse Platform from Microsoft

This image below shows Microsoft's recommended configuration for a BI and Analytics integrated storage solution. Azure Synapse Analytics is the new thing in town, the new MS SQL Data warehouse. The most expensive thing in this configuration is the Azure Data Factory. Our institution already has an FME Server. The question is can we replace Data Factory for FME? and has anyone in the community have made successful implementation of FME in this type of scenario?

Create attributes in multiple .dwg files (not blocks)

Dear fme users, I'm here to ask your most valuable help. I have hundreds of .dwg files in a directory and I need to process all of them adding 6 attributes to each drawing, the values of these attributes must be taken from the file name. I managed to build a workflow that assigns the attribute values to 6 columns but I can't figure out how to convert this values as recognized attributes by the output file produced with a standard writer.Small example:Input file:name: ATTRIBUTE1_ATTRIBUTE2.dwgno attributesOutput file:same nametwo attributes: number one - ATTRIBUTE1 number two - ATTRIBUTE2Thanks a lot for your help.

244

4 years ago

oliver.morrisContributor

asked in Data

sql server jdbc writer slow

HiI am trying to write to sql server rds on amazon using the jdbc writer, it is really slow. What options do I have to make this any faster. Currently it is inserting 1000 records to a 4 column (all strings) table @20s intervals. using sql server import export wizard is about 50x faster.Thank you

3513

4 years ago

Page 80 / 272

Badge winners

nick_ahas earned the badge Welcome Back
nick_ahas earned the badge FME Form Basic Training
j.botterillhas earned the badge FME Flow Certified Professional
steveoladayiyehas earned the badge Integrate Spatial Data
vertiroyhas earned the badge Integrate Spatial Data

Show all badges

30,112: Posts

Terms & Conditions Cookie settings

Sign up

Already have an account? Login

An FME Account is required to contribute

Login to the community

An FME Account is required to contribute

Enter your E-mail address. We'll send you an e-mail with instructions to reset your password.

Enter your e-mail address

Back to overview

Scanning file for viruses.

Sorry, we're still checking this file's contents to make sure it's safe to download. Please try again in a few minutes.

This file cannot be downloaded

Sorry, our virus scanner detected that this file isn't safe to download.

A product of

Company Products Giving Back Careers Contact Privacy Feedback

Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.