- - Community Resources
Ideas
Product Updates
Events
Blog

Home
Forums
FME Form
Data

Data

Using readers and writers to extract and load data.

6,814 Posts
25,095 Replies

When you subscribe we will email you when there is a new topic in this category

6814 Posts

How can i read csv file daily?

Hello, I am a new FME user. im gonna store csv file in one folder daily. (ex. 20180914.csv)If it comes in daily, is there a way to read only today's csv file?Thanks in advanced.

4 years ago

virre

asked in Data

Merge PDFs in a folder

Hi! I've tried looking for a solution for this but can't really get it to work.I have a folder with several PDF files that I just want to merge into one. Ideally I'm looking to specify that folder and then let FME do the job and spit out a Merged.pdf.Is there a simple way to do this? Tried a solution, but several attributes from the original PDF were lost in the translation. I want this to work with multipage input PDFs.I someone has a good solution to this and can give an example .fmw it would be appreciated! Thanks :)

newbie trying to read multiple spreadsheets

I'm trying to read multiple spreadsheets and write one worksheet from each of them into a single output spreadsheet but am going round in circles...Input data: I have a parent directory with an unknown number( and names) of subfolders. Each subfolder has a spreadsheet, results.xls. Each spreadsheet has, say, 10 worksheets, ws1, ws2...ws10.Output: fme needs to search the parent folder to discover the subfolders. Then, for each subfolder, it needs to read the results.xls spreadsheet. A single spreadsheet needs to be written to disk containing worksheet ws5 from each discovered spreadsheet with each worksheet given the name of the subfolder that they came from.The FME excel reader seems to need to know the name of a spreadsheet in advance, so I used a Path reader to discover the subfolders but passing these to a featureReader doesn't seem to read in the spreadsheets properly.Any help would be appreciated.ThanksMacsmith

Update SDE feature class with values from Excel table

Hi all, This should be really simple, but I can't work out what I'm doing wrong in my workspace. Any help appreciated...I need to update a couple of attribute fields in a SDE feature class with values from an Excel table (attributes only, no geometry. Yet.). Using the Excel and 'ESRI Geodatabase (ArcSDE Geodb) readersI'm using the 'FeatureJoiner' transfomer to join the two readers together based on a unique identifier (ESRI Global ID field in the SDE feature class, which the person who created the Excel table also used to capture their new data), then put it through the 'Attribute Manager' transformer so I can map fields between the Attribute Manager and my writer (the columns in the Excel file and SDE feature class have slightly different names, but I know which ones correlate)I'm using the 'ESRI Geodatabase (ArcSDE Geodb) writer, settings:Feature operation: UpdateTable handling: use existing Update spatial columns: NoRow selection: Match column

Parameters in input filename?

I am trying to setup a job in FME that reads a CSV file where the filename changes daily. The filename looks like this HHSVENDORS_02242020.txt. The HHSVENDORDS_ is always the same, but then it has the date appended in the format MMDDYYYY. Is there a way to dynamically create/read the input filename on a CSV reader? So when the job runs each evening, it will pull the correct file.

2310

4 years ago

syarlagaddaContributor

asked in Data

Published to AGOL from geodatabase(one layer) around 400000 records keeps timing out

4 years ago

lcvec

asked in Data

Excel reader parameters issue

Hello. So I'm currently having an issue with the excel reader and I'm not sure I'm missing something. If I drag and drop or add the dataset to the workspace and configure it, it will read correctly. In this case, I want it to start reading the data from row 6 (1 to 5 are the header) and go from there. This works for the initial file, but if I change the source data to pick up a different file with the exact same format, the settings are somehow reset and it starts reading from row 1, resulting in no attributes or features being read. I've tried updating the reader feature types and changing the Microsoft Excel Parameters for it, and while it works when inspecting the data, when I run it, it goes back to missing. What I need is to be able to feed different datasets into this reader but all with the same format, so there's no need for it to be dynamic. However I can't understand why the parameters only work for the first file it reads, despite it being fine when I

How do I read AutoCAD text rotation which is not numeric?

I have been provided with a set of dwgs which have autocad text in it and rotation values expressed in degrees e.g. '96d'. I need to read the value as '96', how do I do this? This is the first time I've seen the rotation units expressed this way. When I translate the dwgs the rotation values do not translate into text or numbers. I'm converting the CAD files into Esri fgdb feature classes and have AutoCAD 2019.

4 years ago

deanhowellEnthusiast

asked in Data

PDF2D writer: invalid map key

Does anyone have any idea what this means and how to fix it? PDF2D writer: invalid map<K, T> key It has something to do with fonts when trying to read from a local font directory. If I clear the true type font folder, I don't get the error but the output PDF is not correct.

Backing Up AGOL Relationship Classes

Hello,Is it possible to automate the backup of ArcGIS Online feature services and associated relationship class(es) within AGOL with FME?I have a current workflow that backs AGOL data both within AGOL and locally as a file gdb; however, the relationship class is never replicated in either backup. I understand for file gdb's that "Relationship classes cannot be created through FME and must be set up through ArcGIS prior to running the translation." However, I was wondering about the AGOL backup data. Any way to retain that relationship class using FME when backing up (writing to) AGOL?

Decoding raw NMEA AIS with other message types

I have files with raw NMEA AIS messages that I need to decode to a delimited text format. Sample raw messages:\\c:1581638400,n:24896*14\\$ANZDA,000000.00,14,02,2020,00,00*79\\c:1581638400,n:24897*15\\!ANVDM,1,1,,A,15Nh=bP00hqWTC:E<3F<6Ign20Sd,0*1C\\c:1581638401,n:24898*1B\\!ANVDO,1,1,,Y,E1IDWe0:aPQRg`94QRagV0W24W3tkNcA:UujH00000P020,4*09\\c:1581638401,n:24899*1A\\!ANVDM,1,1,,B,15N8ct0P00IWliDE;UJpMOv42<1:,0*2C\\c:1581638401,n:24900*1B\\!ANVDM,1,1,,B,15N3Ib?P01IWiupE<espU?v40D0t,0*6B\\c:1581638401,n:24901*1A\\!ANVDM,1,1,,B,15O0JF?013qWbjPE;VwUKDP00<1C,0*3D\\c:1581638402,n:24902*1A\\!ANVDO,1,1,,A,E1IDWe0:aPQRg`94QRagV0W24W3tkNcA:UujP00000uP20,4*4C When I try to read the NMEA strings as AIS messages, the header isn't recognized by FME as a valid AIS message. The documentation lists the only acceptable message type headers as "!AIVDM" or "!AIVDO". I've been told the messages in my data are valid position reports, and they are correctly encoded wit

1700

4 years ago

nagContributor

asked in Data

Arc SDE 32 and FME server 64 bit with 32 and 64 bit engines

Hello Team, Is arc SDE32 bit writer supports with FME Server 64 bit?, We have Arc SDE 10.1 with 32 bit and FME server 64bit with 32 and 64 bit engines installed in the same machine. Is it valid configuration to run a workspace?Thanks...

How to read simple xml

Hello,I know (hope) that there is probably an easy solution for this. I'm trying to read the XML file using the Feature Paths configuration type but it seems I can't get it right... I have simple xml file like this:<A> <ATT1>10</ATT1> <ATT2>20</ATT2> <ATT1>30</ATT1> <ATT2>40</ATT2> </A> I want to read it like this:ATT1ATT2 10 20 30 40 Instead, I'm getting this:ATT1 ATT2 10 <missing> <missing> 20 30 <missing> <missing> 40 Can somebody please tell me what setting I have to apply in XML reader. Thanks!Mario

4 years ago

aquamarine

asked in Data

Interpolation method for not preserving curves in Microstation DGN Reader

Hi,I have been using the Microstation DGN Reader for many years. I always make sure to deselect the 'Preserve Curves' option in the Reader Parameters as this then creates extra vertices to preserve the shape of the curve as per the explanation in the Help page:This parameter controls whether or not curve elements will be stroked into lines by adding vertices.When selected, the control points of the curve are connected into a line. This option preserves the original control points and the end slopes. This is primarily used for DGN-to-DGN translations.When deselected, the control points are used to interpolate a linear representation of the curve. It strokes (vectorizes) the curve (that is, it preserves the shape of the curve). Deselecting this checkbox enables the parameter Number of interpolated curve vertices. However this explanation does not state the interpolation method used for 'vectorizing' the curve. Does anyone know what method is used in FME Desktop 2019.1?

4 years ago

jackydContributor

asked in Data

ERSI ArcGIS FileGeodatabase default attribute values

Hi I ma writing to an exisitng ArcGIS file geodatabse which is set up with subtypes and default attribute values. When i wrtie to table File GDB only the attrbutes specfies with in the workspace a filled in, none of the default values for the other attributes are applied to the features. How do i get the default values to be filled in when i write to the tables, without setting up the values with attribute creator? There over 700 attributes to setup maunally if i cant get the values embeded i nthe fileGDB to write. Or how can i set up the valuies in my workspace with out having to manually type out or cut & paste all 700+? Thanks in advance Cheers Jacky

4 years ago

sandervanegmondContributor

asked in Data

How do I prevent FME from correcting my gml data in the reader?

I have made a workspace which is used to map geometric errors. My data is stored in the gml-format.In my workspace I have a FeatureReader, which reads GML data. Issues in my source data can be among other things:Consecutive vertices in lines or polygonsToo many coordinates in my point elementsNon-closing polygons (last vertex isn’t the same as the starting vertex)The problem I have with my gml FeatureReader is that these errors are fixed automatically by FME. The consecutive points are removed, the last coordinate of my point element is used and my polygon is closed automatically by adding the first vertex as last coordinate. This is convenient when I want to work with the data without hassle, but in this case I want to know what those issues are so I can correct my source data or report errors.Is there any way to show the errors corrected by the FME reader or prevent the reader by doing so?

4914

4 years ago

deanhowellEnthusiast

asked in Data

Adding Arrowhead to PDF Output

We have a DWG drawing where the arrow heads have been created as a block but the arrow heads are not coming into FME and hence I am not able to output the arrow heads to a PDF. I have exploded the blocks into entities but still no luck. Is there a way to recreate the arrow heads when outputting to PDF or bringing the data in from the DWG.Attached is a sample drawing to help understand what I am trying to achieve.Any assistance would be appreciated.

Can the WLAS reader be used to create a dynamic schema?

I am currently attempting to use the las file reader in FME 2019 on multiple files. Unfortunately, the files are not all the same schema. So, I was wondering if it was possible to read these into a dynamic schema? I have attempted to create a dynamic workspace with the reader and when attempting to choose the feature types to read the following error is received: "SOURCE_READER statement must be of the form 'SOURCE_READER <reader type> [<parm> <value>]*' -- `SOURCE_READER WLAS SCHEMA_MODE FEATURE_SCAN -_ENCODING' is invalid : Row came from between lines 1 and 7 of file C:\\Program Files\\FME\\metafile\\WLAS.fmf Failed to read schema features from dataset <path> using the 'wlas' reader"

4 years ago

damaynard01

asked in Data

MS Access app converting to web app

I have a distributed MS Access app with linked tables that has grown to big and has too many users. We need to re-write into a web app. There are many tables, queries, forms, reports etc. so I expect this will not be a quick process. I'm interested in any advice offered on this undertaking. As a first step, we are moving the DB to SQL Server. The reports will be converted to a standard reporting technology like Business Objects or Tableau using the SQL Server. We'll use Python or Java for backend DB connectivity. I'm hoping there is an easy way to convert the UI into HTML pages. Several pages in the Access app consist of forms with drop down list boxes, radio buttons, check boxes, etc. Any advice?

150

4 years ago

solyContributor

asked in Data

How could I look up in Oracle non spatial and extract the data

Could I ask ,How could I look up in field names in Attribute A in oracle no spatial and extract the row of these field names that what am i looking for.I would like to compare between Excel file as reader and oracle non spatial reader and if some field names in Attributes A is missing in excel file so I would like to extract the missing from oracle non spatial And another case ,compare between excel file and oracle no spatial If It finds one field name in excel file is extra sure so I want to look also for this field Name in oracle and extract the row that include this field But I want to say ,excel file and oracle non spatial have one common attribute and the rest of attributes are totally different but I would like to search for this field name and extract the row that relates to this field and put new data in another sheet has the same attributes as oracle..FME version 2018

4 years ago

anders_d

asked in Data

Batch Deploy won't write to ArcGIS online

When trying to Batch Deploy my simple Geopackage files to AGOL the 'Next' button never gets possible. It wants a path to the files to be written. Is it not possible to use Batch Deploy to upload to AGOL or what am I missing?

ECW Writer - ECWJP2 SDK memory allocation

When I try to write an ECW file of around 1 GB, I get the following error from the ECW writer:ECW writer: The ECWJP2 SDK only has 200.0 MB of memory allocated to it and this is not enough to compress this dataset. Please set the SDK's memory limit above 203.9 MBThat sounds fair enough. Does anyone know how to allocate more memory?Using FME Desktop 2019.2.

830

4 years ago

djmcdermottContributor

asked in Data

OAuth 2.0 407 Proxy Authentication Required

Hi FME'ers,I am having trouble authenticating any OAuth 2.0 web connections with FME Desktop 2018.1 (Build 18528). I am trying to access ArcGIS Online but this problem occurs with any OAuth 2 service. I understand that the issue is with my organisations proxy/firewall but I am trying to diagnose precisely where the failure is occurring. FYI, We use a pac file proxy.In Options > Network Proxy, I have 'Use system proxy settings' checked. I know I can get access to the internet with this setting because I have access to FME Hub and can use HTTPCaller. In Web Connection > Manage Services, I select the new connection I created for AGOL and click Test. The credentials screen appears so I enter my credentials and click Sign In. I then get a Network Error dialog box saying "Failed to obtain refresh token: Error retrieving access token. Check Log Window for more details." In the log file I get the message: "407 Proxy Authentication Required" followed my a w

How do I make FBX files from LAS files (with RGB values)?

I have a folder with 92 LAS files, see screenshot with example of one LAS file: .Which transformers and non-default settings are required to make these files into one or more FBX files? A direct reader/writer translation results in an empty FBX file.Any help/suggestions/examples is much appreciated!

Extracting HTML text on scrolling page

4 years ago

Page 84 / 273

Badge winners

lmo_rotthas earned the badge Welcome Back
mathieuhas earned the badge Welcome Back
fmeserver2022+has earned the badge Welcome Back
matt.stullhas earned the badge Expert (Grey)
trembathphas earned the badge Welcome Back

Show all badges

30,384: Posts

Terms & Conditions Cookie settings

Sign up

Already have an account? Login

An FME Account is required to contribute

Login to the community

An FME Account is required to contribute

Enter your E-mail address. We'll send you an e-mail with instructions to reset your password.

Enter your e-mail address

Back to overview

Scanning file for viruses.

Sorry, we're still checking this file's contents to make sure it's safe to download. Please try again in a few minutes.

This file cannot be downloaded

Sorry, our virus scanner detected that this file isn't safe to download.

A product of

Company Products Giving Back Careers Contact Privacy Feedback

Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.