Skip to main content

Data

Using readers and writers to extract and load data.

  • 6,932 Posts
  • 25,508 Replies
6932 Posts
A
Anonymous
asked in Data
Save each record to file

Do the FME text / csv writers have the ability to save each record that gets sent to them? I am running a process that will retrieve about 200,000 records. From my test results it appears that FME only writes to the file when the process completes. But since this process will take a few days (I have a decelerator in the script) I don't want the script to choke part-way through. It would be better to save each record that arrives at the writer. Thanks

190
pratap
Contributor
O
ola
asked in Data
Writing multipatch features to Sketchup groups

Hello, I have a number of 3D-buildings saved in a multipatch layer in an ESRI file geodatabase. These buildings are saved with unique object ID-numbers and a few other attributes. I need to create a FME workspace that reads these objects, and outputs them as object groups in a skp-file (local coordinate system, Sketchup 2013). I have created a workbench that reads the multipatches and writes them to their own layer in a skp-file using the GeometryPropertySetter and a attribute called "sketchup_layer_name" (="Buildings"). But, instead of being written as sketchup groups (i.e. a group of surfaces that equals a multipatch), each surface from the multipatches are being written as separate faces. I am able to select all the faces that belong to a certain building, rightclicking them and clicking "Create Group", but since I am working with severeal thousand buildings I need a way to automate this process. Any help is greatly appreciated :).

340
gio
Contributor
K
kitty
asked in Data
Does fme server 2014 work with fme desktop 2012?

Can FME Server 2014 work with workspaces created at FME Desktop 2012?

70
david_r
Evangelist
M
mani
asked in Data
Problem in reading XML file

Hi, I am very new to xml and when I want to open my xml file, I have a reader in navigator but no feature types appear. In log file I receive this message: <R_1 Reader> - No GML features were found in the set of XML Schemas, make sure that at least one of the schemas is a user, domain or community-specific "GML Application Schema". UniversalReader -- readSchema resulted in 0 schema features being returned Reader Parameter(0) = >C:/Users/TEMP.GIS.000/Downloads/codelists-1.0-examples/codelists-1.0-examples/BuildingFuctionType.xml< Reader Parameter(1) = >MAP_EMBEDDED_OBJECTS_AS< ....... UniversalReader -- readSchema resulted in 0 schema features being returned .... What I need in my workspace is to have the building types and their values for further work. <dictionaryEntry> <gml:Definition gml:id="id21"> <gml:description/> <gml:name codeSpace="urn:d_nrw_sig3d">1000</gml:name> <gml:name>residential building</gml:name> </gml:Definition> </dictionaryEntry> I don't know how to configure the r

194
M
T
t
asked in Data
How do I create a smallworld dataset from shapes files?

How do I create a smallworld dataset from shapes files?

10
T
jess
Contributor
jessContributor
asked in Data
Remove feature dataset when writing to ESRI file-geodatabase?

Hello! I'm curious if there is a way to strip away the feature dataset information when using an ESRI file-geodatabase writer (GEODATABASE_FILE, dynamic mode)? My workbench reads a series of layers from an SDE database, performs some geoprocessing and then writes all the data to a new file geodatabase. I'd like to remove the feature dataset names which come from the SDE and output the layers in the root of the fGDB. Any suggestions? Thanks in advance, ~jess~

1023
jess
Contributor
A
alison
asked in Data
Mapinfo (MAPINFO) writer forcing to v12

I am having a problem similar to https://safecommunity.force.com/CommunityAnswers#!/feedtype=SINGLE_QUESTION_SEARCH_RESULT&id=906a0000000Ce4vAAC When recently using FME 2014 i must have unconsciousely switched from using the Mapinfo (MITAB) writer to the new MapInfo (MAPINFO) writer. I recently discovered that files i had created would not open in version 11.5 of MapInfo, giving me a message saying that it would only open in v12. This morning I discovered that using the MITAB writer works ok. This is important as I deal with people working on versions 7 - 12. Please can someone explain the difference. I have never known a TAB file to be fussy about what version it was created in before. I can only find documentation on the difference between the MFAL and MITAB writers Thanks

70
A
tono
Contributor
tonoContributor
asked in Data
Local path from Desktop to Server

Previously you could choose for instance C:\\temp\\input.pdf on FME Desktop and then tell FME Server that this should NOT be changed (ie, the files are still located at c:\\temp\\input.pdf) on the Server. With FME Server2014 I can not seem to get this to work. Even when using the DO NOTHING parameter on the Manage resources when uploading. Is this a bug? What is the workaround? FME Server 2014 - Build 14235 - 32 bit win, same FME Desktop version, 32-bit

110
tono
Contributor
H
hr
asked in Data
optimizing memory usage

Hi, I have a large amout of data in csv files 4 mil in each. Always gets memory problem when reading the data. The rest of the process is fine. I use a 64 bit fme on a powerful machine and the memory is only on 60%. Any suggestions? Henrik Rinne

201
david_r
Evangelist
bubblebeb
Contributor
bubblebebContributor
asked in Data
limit CSV writer size

I have a CSV output file which is currently 54mb. I need to upload it in a system which has a limit of 5mb. Is there a way of limiting the writer to 5mb limit and then write to another CSV. i.e. I want to get 11 csv's of the same data. Ideally with a counter in the name i.e. upload1, upload2, etc. Is there a way of either setting a writter parameter or a transformer to do this?

780
bubblebeb
Contributor
M
mani
asked in Data
CityGML - ADE - problem in nested feature attributes

Hi all, I have a question. and one problem! I have a CityGML file with lots of generic attributes. On the other hand I have CSV file containing all these generic attribute names and their values. What I need to do is a validity check for these generic attributes. Like if all obligatory attributes are given as generic attributes and if the assigned values are in the range. My question is: Is it possible in FME to filter feature's attributes based on attribute names like having only those attributes which do not contain citygml in their names?! My problem: I am going to write a citygml file into CityGML ADE-inspire. (I have the schema for the ADE) My problem is regarding the nested feature attributes. I mapped every things correctly and when the output is redirected to FME inspector the results are OK but when I need to write it in a gml file the inspire attributes (list ones) are missing. Example: In attribute creator I have: core3d_inspire_id{0}.base_identifier_base_namespace core3d_in

70
M
jfd553
jfd553
asked in Data
PostgreSQL connection: Unknown type with OID xxx

I am writing features in a PostgreSQL database. Everything seems OK except for the following warning... INFORM|Truncating existing table 'public.user_locations'... INFORM|Appending to existing table 'public.user_locations'... WARN |PostgreSQL connection: Unknown type with OID 24635, treating as char(-1) What does that warning means and what should I do to get rid of it? Since I am about to load hundreds of GB of data, I just want to be sure everything will run smoothly after the upload...

463
david_r
Evangelist
D
djabx
asked in Data
EsriReprojector FME Server

I've made a workbench witch project some geometry with EsriReprojector. It work very well on my desktop (FME 2013 SP4 build 13547 WIN32) but on the server (FME Server 2013 build 13547 win32 on a Window 2008 R2 Enterprise x64) it crash without any log (with the exact same geometry). Any help will be appreciate !

100
D
G
george
asked in Data
can an FME server host an OGS service?

Can an FME server host any map services (e.g. OGC's wms services)?

41
david_r
Evangelist
B
buniekwawa
asked in Data
access mdb - Provider error `(-2147467259)

Hello, i've have problems with reading some access mdb files. I have win7 x64, office x32, fme x32 when i try to read mdb received from client, i get error: MS Access Reader: Connection failed. Connection string `Provider=Microsoft.Jet.OLEDB.4.0;Jet OLEDB:Engine Type=5;Data Source=C:/mbu/database.mdb;User ID=admin;Password=********'. Provider error `(-2147467259) Unrecognized database format 'C:/mbu/database.mdb'.' I can manualy open mdb in access and save as accdb, then it will read in fme workbench, but i'm going to do it many times, so i'll be greatfull if someone can provide solution. i've tried reinstalling fme, access database engine - didn't help.

150
B
J
johng
asked in Data
Excel Reader

Hello, I would like to read excel files containing one sheet only. But columns headers may not be same apart from 2 columns Lat and Long. I am using VertexCreater to create point from Lat,long. e.g. Excel1.xls contains A,B,C,Lat, Long Excel2.xls contains C1,C2,C3,C4,Lat, Long,C5,C6 Excel3.xls contains Lat,Long,A1,B1,C1 How can I read excel file with dynamic scheam and retain all attributes for destination format. Cheers.

161
takashi
Contributor
jamal
jamal
asked in Data
Choosing parameter value from a list stored in an Excel field,

Choosing parameter value from a list stored in an Excel field, I couldn’t figure out how the a parameter value can be selected from a list saved in an Excel table For example, in the screenshot below, I wanted to choose the “PlanType” value from the shown Excel table How this might be accomplished? Thank you Best Jamal

42
jamal
J
johng
asked in Data
FME KML file name (bit strange behavior)

Hello, I have observed that FME behaves differently if we run a workspace in Workbench and after publishing to server. I have a workspace with Shape Reader reads multiple files and converts to KML using GENERIC writer. If I fanout the dataset on fme_basename it works perfectly in workbench, but if same workspace is published to server and run as service it creates output kml as "output.kml" e.g. Shape reader reads Test1.shp & Test2,shp Creates output by fanning out on fme_basename Output KML : Workbench creates Test1.kml in Test1 folder & Test2.kml in Test2 folder - PERFECT On server : "output.kml" in Test1 folder and "output.kml" in Test2 folder Could anybody explain this behavior? Cheers John

130
davideagle
Contributor
G
george
asked in Data
How to publish workspaces to FME server without using FME desktop

It is encouraged to automate the deployment proces for production in our organization. Deployments are usually done by designated people especially for production development. Can you give me any suggestions on automatically publish workpsaces (many of them) to FME servers? like using scripting, .......

110
davideagle
Contributor
F
frost
asked in Data
Calculate builiding elevation from lidar

Hi, I'm looking to calculate a building elevation and output the attribute to an sdf. What I have is xyz files for bare earth, xyz of full feature (includes all elevated points) and an sdf of building footprints. I need to somehow get the difference between the full feature point elevation and the bare earth elevation, then clip out all points that are outside of the building footprints, and lastly create the building footprint sdf. I've never done this in FME before, so it will be a new process. thanks.

391
davideagle
Contributor
jelle
Contributor
jelleContributor
asked in Data
Views Oracle to SQL Server

Fellow FME'ers, we have migrated data from Oracle Spatial to SQL Server. It works like a charm, without any transformer in between. Does anybody have an idea how we could convert the views, defined in Oracle, to SQL Server using FME? best regards, Jelle

230
gio
Contributor
G
george
asked in Data
FME engine licenses vs. ArcGIS licenses

Hi All, If I have two engine licenses, does it mean I need two ArcGIS licenses in order to run jobs (connecting to SDE) in parallel (make full use of 2 engines)É Thanks, GS

00
G
jamal
jamal
asked in Data
Accessing published workbench from client machine,

Accessing published workbench from client machine, I have published simple workbench that converts CAD files to shape files but couldn’t figure out how this workbench can be used from the client side. The link below doesn’t work from the client side http://10.110.62.82:8080/fmedatastreaming/Samples/CAD_TO_Shape.fmw?SourceDataset_GENERIC=%24(FME_MF_DIR)Ayman_02.dwg&FEATURE_TYPES=Parcels_Geometry&DestDataset_SHAPE=D%3A%5CS1 What might be the issue here? Thank you Best Jamal

50
jamal
jamal
jamal
asked in Data
The ArcGIS FME keeps the old local IP,

The ArcGIS FME keeps the old local IP, We have switched the local IP to real IP. It appears that the FME Servers still generating URLs with the Local IP. What might be the issue here? How to set the FME Server to generate URLs with the Real IP? Thank you Best Jamal

40
K
gschleusner
Contributor
gschleusnerContributor
asked in Data
Postgre Read and Write SSL

Is there a way to connect to a Postgre server using an SSL connection? Thanks, Greg

160
fmelizard
Contributor

Most helpful members this week

Badge winners

  • Student (Grey)
    jferrerhas earned the badge Student (Grey)
  • Welcome Back
    mapianhas earned the badge Welcome Back
  • Student (Grey)
    icemansernahas earned the badge Student (Grey)
  • Expert (Grey)
    pauldeqhas earned the badge Expert (Grey)
  • Welcome Back
    ayrajuhas earned the badge Welcome Back
Show all badges
30,851
Posts
Terms & ConditionsCookie settings

Cookie policy

We use cookies to enhance and personalize your experience. If you accept you agree to our full cookie policy. Learn more about our cookies.

 
Cookie settings
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.