Data

Using readers and writers to extract and load data.

  • 6,443 Posts
  • 23,829 Replies

6443 Posts

helmoetz
Contributor
helmoetzContributor
asked in Data

Postgis varchar() interpreted by FME as varchar(80)

Hi, I have this dynamic setup where a postgres FeatureReader reads the schema and the results are written to another database using this schema information to create the new tables. However, a table having colums of type unqualified varchar will be regenerated with varchar(80) and if records are loaded having length >80 the workspace will crash, saying: Error was 'ERROR:  value too long for type character varying(80)I attached the workspace. Is this a bug in FME or am I doing something unbelieveable silly?😕 I'm using FME(R) 2020.2.3.0 (20210129 - Build 20820 - WIN64) on a postgres 12.9 (Ubuntu 12.9-1.pgdg18.04+1). 

3
felipeverdu
Contributor
1 hour ago
benicholls
Participant
benichollsParticipant
asked in Data

Parse JSON from Bentley OpenGround

I would like to be able to parse a response from the Bentley OpenGround REST API, and am getting somewhere but not far enough!I can use a mixture of attribute exposer and JSONFragmenter to get some information out, but it never quite looks like the table i want. i will be doing multiple queries over the coming months into this api, and will have a different number of columns each time, so cant parse through each data record based on the number of columns i want in that particular query! I can get to the point where i have a feature for every feature in the database, and then i have a _response_body which i need to parse further. i can keep parsing it through jsonfragmenter, and i end up with a jumbled mess where columns are names wrong and the colunn information i want is in the wrong part of the table. it is a simple structure, such as “DataFields” “heade” for column name and “value” for data itself. i will add some screenshots to aid in query. any help would be greatly appreciated!Th

1
nielsgerrits
Celebrity
21 hours ago
M
mradamlenz
asked in Data

Projecting Geotiff Raster to Maps - 101

Hi All,I'm new to FME and have been pouring over videos to try and do my first projection. Any help is much appreciated. I have a raster geotiff I'm trying to project onto a City map. I'm hoping someone can point me in the right direction with a start to finish video or thread to help guide me through the process.Thanks,Adam

1
dgarshadali
1 day ago
palhermans
Contributor
palhermansContributor
asked in Data

FME2024.0 "Esri ArcGIS Layer" Reader failes when LYRX from ArcGIS Pro includes Group Layers

Hi. reading LYRX (“layer files”) from ArcGIS Pro failes when the layer file include Grouped Layers. It works just fine if the LYRX file does not include Group Layers.Thea Layer File/LYRX is made with ArcGIS Pro 3.2.2 where the LYRX file is in JSON format.

3
palhermans
Contributor
1 day ago
pauld
Contributor
pauldContributor
asked in Data

Looping through string to remove known text

Hi.I have another string replacer question.I have a file of text strings (appellation) similar to LOT 1/2 DP 17528 LOT 1/2 DP 13552 RS 16932 18812 19986/7 PT 18813 20862  BLK IV KAPUNATIKI SDI have split the LOT 1/2 DP 17528 & LOT 1/2 DP 13552 into 4 separate rowsTemp, _result, appellationLOT 1/2 DP 17528, LOT 1 DP 17528, appellationLOT 1/2 DP 17528, LOT 2 DP 17528, appellationLOT 1/2 DP 13552, LOT 1 DP 13552, appellationLOT 1/2 DP 13552, LOT 2 DP 13552, appellation What I want to be able to do is remove the temp part from appellation  The resultant appellation in this example would be: RS 16932 18812 19986/7 PT 18813 20862  BLK IV KAPUNATIKI SDI have tried doing this but only ended up removing one part from each instead of both Temp parts. I would be very grateful for any help with this.Regards, Paul

3
pauld
Contributor
1 day ago
gisbradokla
Contributor
gisbradoklaContributor
asked in Data

dynamic crs to separate data stream

I am reading datasets and can extract the crs (stateplane) from them. but I need to use a spatialrelator for data that is in WGS. I need to be able to determine incoming crs and set a reprojector on the wgs data right before the spatialrelator. In the past i have only been able to accomplish this by stringing together more than 1 workbench.Any ideas on how to get this done in 1?

4
gisbradokla
Contributor
1 day ago
bayram
Contributor
bayramContributor
asked in Data

CITYGML EXPORT

Hello,I created a model and I want to export it as citygml, but the model does not appear in the FZK gml viewer. How can I solve this problem? Am I making a mistake in the writer? I added my data and FME WB. Could you please check it and share what I am missing?

2
bayram
Contributor
2 days ago
alfons
Contributor
alfonsContributor
asked in Data

change coordinate format DM -> DD

Hello, I'm sure it's a pretty simple question, but somehow I can't see the wood for the trees to find the answer.I have a list (.xls) of coordinates in the form <name>, <x_coord>, <y_coord>. The coordinates are, for example, in the form N 51° 06.796 E 3° 14.545.Now I want to save the information in an FGDB and would have expected the VertexCreator to help me with this. Unfortunately, all my values are completely rejected. My further attempts - removing all spaces; using the HUB Transformer "DegreeFormatter" or Decimal Degrees Calculator all failed. For this reason, how do I manage to get the coordinate format into a form so that the VertexCreator accepts it? Thank you very much.

1
virtualcitymatt
Influencer
2 days ago
F
fdh12316
asked in Data

How can i change the unit of an IFC file to meter

Hi there,I have a question about the unit of coordinate in the iFC file, which i wanna clippe with another solid. The coordinates are 1000 times bigger, and the clipper could thus not find overlaps. How can I adjust the coordinate value: using the expression evaluator, and divide all the coordinates with the value 1000? if tho,How can I access all my coordinates because I could not find an attribute that enables that? I also tried the coordinate extractor but it is only another copy of the coordinate values, which doesn't update the geometry coordinate value in the IFC . If anything i understand it wrong pls point it out! thanks in advanceFang

7
stormy
2 days ago
xingfu07
Contributor
xingfu07Contributor
asked in Data

Adding in a custom coordinate system

Hi All, I’ve tried to add in a custom coordinate system and it just doesn’t work. I’ve restarted my computer twice and the co ordinate system won’t come up in the csmap reprojector. I would love to use the online webform that has been created, but I am unsure at which username and password is required to log-in. My usual FME doesn’t work. I’ve created a file in the FME folder in documents and edited both coordinate files in the reprojector folder. This is what I have written below:# COORDINATE_SYSTEM_DEF PerthCoastalGrid1994     \\#      PROJ TM                                  \\ #      UNIT METER                               \\#      SOURCE "spatialreference.org"            \\ #      DT_NAME GDA94                            \\#      EL_NAME GRS1980                           \\ #      DESC_NM "Perth Coastal Grid 1994"        \\ #      GROUP AUSNZ                              \\ #      PARM1 115.8166666666667                  \\     #      ORG_LAT 0                                \\  

2
xingfu07
Contributor
2 days ago
jakethepainter
Contributor
jakethepainterContributor
asked in Data

Can fme read whether a cell has comments or notes in an Excel xlsx document?

I have a giant spreadsheet that many people have added notes or comments to by right-clicking on the cell and saying "new comment" or "new note". Is there a way for fme to figure out which cells have comments?

5
jkr_wrk
Enthusiast
2 days ago
seanclarkner
Contributor
seanclarknerContributor
asked in Data

Merge features from Table A using a field value to Table B if the Table A value matches one of a set of comma delimited values in a field in Table B

A have one dataset (Dataset A) that has contains fields, one of which is CityName; example value “Sudbury”. I have another dataset (Dataset B) that contains a field made up of comma delimited AltNames; example value “Gran Sudbury,Grand Sudbury,Sudbury, etc.I need for these features to be joined in one dataset if the value in CityName matches any one of the comma delimited names in AltNames, however different records have different numbers of alternate names and it cannot match with a section of a comma delimited value. For example, Sudbury != Grand Sudbury.I think that lists may solve this but I’m not familiar with the transformers that might enable this process.

2
seanclarkner
Contributor
2 days ago
eric_armitage
Contributor
eric_armitageContributor
asked in Data

OData writing

I am starting to work with reading and writing OData.  I can see there is an OData reader but no writer. Writing seems to be possible using the HTTP caller.  Is there a particular reason why there is no OData writer? Are there plans for an OData writer? Just want to make sure we are building the process correctly.

0
eric_armitage
Contributor
3 days ago
lise
Observer
liseObserver
asked in Data

Visualising data on inspector map after reprojection

I would like to visualise data on the inspector data or in the graphic view after I reprojected it. That would be in the middle of a workspace before using other transformers and just to locate some features. I know it’s possible to write the data in a shape and check the features on another software like QGIS or ArcGIS but I would prefer to do that in FME directly if that’s possible, when I’m building my workspace. Can I do that?

4
lise
Observer
3 days ago
davidlee
Supporter
davidleeSupporter
asked in Data

How do I convert a Raster file to COG (Cloud optimised Geotiff) with 4 bands (R,G,B,Alpha)?

I want to convert JPEG file (4 band) to COG with below parameter:I have problem with “Interleaving Type”:When i chooses “BIP”, i received a message log:“COG writer: JPEG compression supports at most 3 bands with BIP interleaving. Please change the interleaving to BSQ, choose another compression method, or reduce the number of bands”When i chooses “BSQ”, i received a message log:“COG writer: The file layout 'Cloud Optimized Tiles' may only be used with BIP interleaving. Please change the interleaving to BIP or select a different file layout”I need to convert JPEG file to COG with compression method “JPEG” and keep alpha band.Can i do that? Am I mistaken about something?

5
davidlee
Supporter
3 days ago
kersy
Contributor
kersyContributor
asked in Data

Value of attribute could not be converted to type 'timestamptz'.

I am trying to upload ifc data through PostGIS Writer and while doing so, I want to generate and store the time of upload as well. However, it gives the following error: “Value of attribute 'begin_lifespan_version' could not be converted to type 'timestamptz'. Feature will be logged and skipped.” can someone help? I am a beginner in FME as well.  

2
kersy
Contributor
3 days ago
mmccart
Contributor
mmccartContributor
asked in Data

Does FME 2024.0 Support Esri's New Big Integer Field Types When Creating New Dynamic Feature Classes?

I have been using Dynamic writers to create feature classes and tables in an Esri File Geodatabase for years. I normally use the FME field type “fme_int32” to create Long Integer field types. However, I need to create attributes using Esri’s new ‘Big Integer’ field type. I assumed I could use “fme_int64” but when I run my workspace in FME 2024.0, it runs without errors but it doesn’t create the new feature class. When I change the 2 fields from fme_int64 back to fme_int32, it runs without errors and creates the new feature class. I have ArcPro 3.2.2 installed which also supports the new Big Integer (64 Bit) field types.My question is: Should I be using a different FME Field type besides ‘fme_int64’ to generate Big Integer field types via a Dynamic FGDB Feature Class Writer?Thanks in advance.

6
crystalatsafe
Safer
3 days ago
slingeman
Participant
slingemanParticipant
asked in Data

AutoCAD DWG to SVG

I’m trying to convert DWG’s to SVG’s. It works, but it’s using the AutoCAD colors. I would like to just make the lines all one color, and NOT use the autocad colors. Ideally, I’d like to fill certain layers with another color.

1
david_r
Influencer
3 days ago
marias
Contributor
mariasContributor
asked in Data

What can be usedsimilar to FeatureReader transformer to read schema and data?

I need to extract TAB or Shape file from S3 and read data from it including schema.I have discovered in this articlehttps://community.safe.com/data-7/how-to-extract-the-actual-shapefile-from-featurereader-geometry-and-attributes-14548?tid=14548&fid=7that FeatureReader apparently can’t read all attributes that are in the file, until I expose them. But what is I don’t know the list of all attributes? I don’t know what exactly will be uploaded by other people to S3 bucket, I just need to read the file.  Could you help, please, to identify the correct settings of FeatureReader or another way to extract data?

0
marias
Contributor
3 days ago
marias
Contributor
mariasContributor
asked in Data

How to replace FeatureReader transformer?

I need to extract TAB or Shape file from S3 and read data from it including schema.I have discovered in this articlehttps://community.safe.com/data-7/how-to-extract-the-actual-shapefile-from-featurereader-geometry-and-attributes-14548?tid=14548&fid=7that FeatureReader apparently can’t read all attributes that are in the file, until I expose them. But what is I don’t know the list of all attributes? I don’t know what exactly will be uploaded by other people to S3 bucket, I just need to read the file.  Could you help, please, to identify the correct settings of FeatureReader or another way to extract data?

0
marias
Contributor
3 days ago
marias
Contributor
mariasContributor
asked in Data

How to replace FeatureReader?

I need to extract TAB or Shape file from S3 and read data from it including schema.I have discovered in this articlehttps://community.safe.com/data-7/how-to-extract-the-actual-shapefile-from-featurereader-geometry-and-attributes-14548?tid=14548&fid=7that FeatureReader apparently can’t read all attributes that are in the file, until I expose them. But what is I don’t know the list of all attributes? I don’t know what exactly will be uploaded by other people to S3 bucket, I just need to read the file.  Could you help, please, to identify the correct settings of FeatureReader or another way to extract data?

0
marias
Contributor
3 days ago
R
robertr
asked in Data

How to extract the actual shapefile from FeatureReader (geometry and attributes)

Hello, I am using a "Directory and File Pathnames" reader to pull shapefiles from a folder. I then walk them through a tester to filter out the files that I do not need. All good and working up to this point. The goal is to work on those shapefiles and have a processed shapefile as an output, in the end.  After the tester, I send the "passed" ones to a "FileReader" that outputs the desired features under the <Generic> port.  My question is: how can I write those out and keep the geometry + attributes of the shapefiles for future processing?   I can see them in the Inspector but I cannot figure out how to move on to the next step as the only attributes I see imported to any transformer I add, are the ones from the "Directory and File Pathnames" reader, not the shapefiles themselves.Long story-short, I want to use the shapefiles that passed the "Tester" as readers. 

5
marias
Contributor
3 days ago
sar_copp
Participant
sar_coppParticipant
asked in Data

Transposing Using the postcode and question number as a pair, possibly a list?

Morning, I am trying to transpose some data from a survey, I want to sort of Pair up the Postcode and the Question Number, then have columns of Responses with counts. Ive attached images  and a fme workbench of the original data and the output I want. I have 36 questions, so want to know if there is a way to do this using a list? So I dont have to repeat this 36 times, I am not understanding how to do lists!I want to output the postcode as a column, then have a question number column. So each postcode will be repeated depending on the number of questions e.g.Question No    PostcodeQuestion 1      M19 4NYQuestion 2     M19 4NYQuestion 1      M4 7ATQuestion 2     M4 7AT Then I want the next columns to be counts of the answersQuestion No    Postcode  Strongly Agree      Agree      Neither    Disagree     Strongly DisagreeQuestion 1      M19 4NY             2                         3               1               0                      2Question 2     M19 4NY             5               

0
sar_copp
Participant
4 days ago
jesselangdon
Contributor
jesselangdonContributor
asked in Data

Splitting bulk features into individual features when performing simple copy of annotation between File GDBs

I am trying to do a very basic read/write of an annotation feature class from one file geodatabase to another. In my case, I’m first running a PythonCaller transformer to remove any and all annotation feature classes in the target file geodatabase, then I use a FeatureReader to read the annotation feature class from the source file GDB, and then run a FeatureWriter to write the annotation FC to the target geodatabase with “Insert” and “Create if Needed” options selected for the table, which I assume is creating a new annotation feature class in the target FGDB.Regardless of minor tweaks I make to both the FeatureReader and FeatureWriter transformers, FME Flow keeps writing these as individual features instead of bulk features. There are ~450k annotation features in the feature class, so running this by writing individual features takes over an hour, which will not work for our scenario.Interestingly, the first time I ran this in an earlier workspace, FME Flow used the bulk feature sett

3
evieatsafe
Safer
4 days ago
soly
Contributor
solyContributor
asked in Data

how to find duplicate of specific column ?

Hello , i would like to find duplicate of specific column As exampleColumn A contain 256789 - 45674567 - 256789256789 - 4567i would like to find all duplicate values at column A ,consider  the values as duplicated in case  a-b or b-a i mean , 3 above values consider are matched .the target to ignore order of each cell How could I achieve it ?thanks in advance FME 2021  

3
liamfez
Supporter
6 days ago

Badge winners

  • Teacher (Bronze)
    tvahas earned the badge Teacher (Bronze)
  • Teacher (Grey)
    tvahas earned the badge Teacher (Grey)
  • Student (Grey)
    anteboy65has earned the badge Student (Grey)
  • Teacher (Grey)
    anteboy65has earned the badge Teacher (Grey)
  • Expert (Grey)
    crystalatsafehas earned the badge Expert (Grey)
Show all badges
28,731
Posts
Terms & ConditionsCookie settings
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.