Skip to main content
Question

FME is mangling my ampersands!

  • January 27, 2016
  • 5 replies
  • 44 views
J
Forum|alt.badge.img+4

Morning...looking for assistance with FME desktop and its CartoDB writer. I have a workspace which writes several thousand planning application points to our CartoDB account. One field in that dataset is a URL for each planning case linking back to details on respective websites. Quite a few of these records contain URLs which have ampersands separating the query parameters.

If I write the records to CartoDB using the CartoDB uploader transformer the ampersands get mangled into %26amp; This isn't even a correct encoding. Ampersands should either be one of the following...

 %26 OR &  

.....and never a mix of both!! If I write the records out to a CSV file instead everything is fine. If I manually upload the CSV file to CartoDB everything is fine....it just seems to be the CartoDB uploader transformer.

Sample URLs below. The top one is coming out of the CartoDB uploader transformer (thanks to the extra inspection button in FME 2016 this output is easy to see now). The second URL is correctly formatted with the ampersand. Any thoughts on this problem would be welcome.

http://www2.guildford.gov.uk/publicaccess/applicat...

http://www2.guildford.gov.uk/publicaccess/applicat...

0684Q00000ArC0IQAV.jpg

This post is closed to further activity.
It may be an old question, an answered question, an implemented idea, or a notification-only post.
Please check post dates before relying on any information in a question or answer.
For follow-up or related questions, please post a new question or idea.
If there is a genuine update to be made, please contact us and request that the post is reopened.

5 replies

mark2atsafe
Safer
Forum|alt.badge.img+56
  • mark2atsafeSafer
  • Safer
  • 2554 replies
  • January 28, 2016

Hi James. Did you ever get this to work? I assume you mean "writer" rather than "transformer" as I can't find a CartoDB transformer. Or am I wrong there?

I've posted this link to one of our developers, but regardless I think you should file this with our (or your reseller) support team if you still can't get it to work.

Regards

Mark

Product Evangelist

Safe Software Inc

FME Evangelist to the Rich and Famous!!
mark2atsafe
Safer
Forum|alt.badge.img+56
  • mark2atsafeSafer
  • Safer
  • 2554 replies
  • January 28, 2016

Hi James. Did you ever get this to work? I assume you mean "writer" rather than "transformer" as I can't find a CartoDB transformer. Or am I wrong there?

I've posted this link to one of our developers, but regardless I think you should file this with our (or your reseller) support team if you still can't get it to work.

Regards

Mark

Product Evangelist

Safe Software Inc

Our developer says that it is a likely problem with FME, but saying that we've tried and fail to reproduce the problem. Can you contact the support team and include a workspace so we can try and reproduce the issue? Thanks.

FME Evangelist to the Rich and Famous!!
J
Forum|alt.badge.img+4
  • james_rutter
  • Author
  • 18 replies
  • January 29, 2016

@Mark2AtSafe Hi Mark and thanks for the reply. I've still got the same problem with fme fudging the ampersands. I'm using a work around at the moment and writing the datasets out to Google Sheets which are synced with CartoDB tables. That's working fine.

Sterling Geo, our FME suppliers, are monitoring this thread so they can take up with Safe direct if need be. In the meantime I've attached the workspace I'm using. It's pulling in GeoJSON from a web resouce, doing a bit of data massaging and pushing out to CartoDB with the CartoDB writer (You'll need to add your own CartoDB api key and a/c into the writer settings).

Just as an additional note....there's no option to truncate the CartoDB in the CartoDB writer format parameters. It would be extremely useful to have this option as we currently have to use a work around using a sampler and httpcaller transformer to do this. Not a big deal but it could be slicker just having this built into the writer (just like it is with the Google Sheets writer)ampersand-test-workspace.fmw

mark2atsafe
Safer
Forum|alt.badge.img+56
  • mark2atsafeSafer
  • Safer
  • 2554 replies
  • January 29, 2016

@Mark2AtSafe Hi Mark and thanks for the reply. I've still got the same problem with fme fudging the ampersands. I'm using a work around at the moment and writing the datasets out to Google Sheets which are synced with CartoDB tables. That's working fine.

Sterling Geo, our FME suppliers, are monitoring this thread so they can take up with Safe direct if need be. In the meantime I've attached the workspace I'm using. It's pulling in GeoJSON from a web resouce, doing a bit of data massaging and pushing out to CartoDB with the CartoDB writer (You'll need to add your own CartoDB api key and a/c into the writer settings).

Just as an additional note....there's no option to truncate the CartoDB in the CartoDB writer format parameters. It would be extremely useful to have this option as we currently have to use a work around using a sampler and httpcaller transformer to do this. Not a big deal but it could be slicker just having this built into the writer (just like it is with the Google Sheets writer)ampersand-test-workspace.fmw

Thanks. I passed the workspace on and we'll see what happens.

Yes, the truncate option is a known issue. We're working on adding that (fyi it's PR#60754) and I hope to see it soon. I'll add your details so you get informed when it is. Cheers.

FME Evangelist to the Rich and Famous!!
helmoet
Forum|alt.badge.img+8
  • helmoet
  • 195 replies
  • November 27, 2019

Hi @mark2atsafe,@james_rutter,

I ran into probably into the same trouble.

The mangling is caused by the special meaning that the ampersand has for the operating system.

A workaround is to generate a text that has no meaning anymore, say an ampersand followed by a space and replace that superfluous space with no replacementtext.

Like in the accompanying .fmwt which contains an Excelsheet that reproduces the error.

ampersand.fmwt

Community Stats

32,079
Posts
121,808
Replies
39,711
Members

Latest FME

Terms & ConditionsCookie settingsAccessibility statement
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.