Skip to main content
  • EN

Morning...looking for assistance with FME desktop and its CartoDB writer. I have a workspace which writes several thousand planning application points to our CartoDB account. One field in that dataset is a URL for each planning case linking back to details on respective websites. Quite a few of these records contain URLs which have ampersands separating the query parameters.

If I write the records to CartoDB using the CartoDB uploader transformer the ampersands get mangled into %26amp; This isn't even a correct encoding. Ampersands should either be one of the following...

 %26 OR &  

.....and never a mix of both!! If I write the records out to a CSV file instead everything is fine. If I manually upload the CSV file to CartoDB everything is fine....it just seems to be the CartoDB uploader transformer.

Sample URLs below. The top one is coming out of the CartoDB uploader transformer (thanks to the extra inspection button in FME 2016 this output is easy to see now). The second URL is correctly formatted with the ampersand. Any thoughts on this problem would be welcome.

http://www2.guildford.gov.uk/publicaccess/applicat...

http://www2.guildford.gov.uk/publicaccess/applicat...

0684Q00000ArC0IQAV.jpg

Hi James. Did you ever get this to work? I assume you mean "writer" rather than "transformer" as I can't find a CartoDB transformer. Or am I wrong there?

I've posted this link to one of our developers, but regardless I think you should file this with our (or your reseller) support team if you still can't get it to work.

Regards

Mark

Product Evangelist

Safe Software Inc

Hi James. Did you ever get this to work? I assume you mean "writer" rather than "transformer" as I can't find a CartoDB transformer. Or am I wrong there?

I've posted this link to one of our developers, but regardless I think you should file this with our (or your reseller) support team if you still can't get it to work.

Regards

Mark

Product Evangelist

Safe Software Inc

Our developer says that it is a likely problem with FME, but saying that we've tried and fail to reproduce the problem. Can you contact the support team and include a workspace so we can try and reproduce the issue? Thanks.

@Mark2AtSafe Hi Mark and thanks for the reply. I've still got the same problem with fme fudging the ampersands. I'm using a work around at the moment and writing the datasets out to Google Sheets which are synced with CartoDB tables. That's working fine.

Sterling Geo, our FME suppliers, are monitoring this thread so they can take up with Safe direct if need be. In the meantime I've attached the workspace I'm using. It's pulling in GeoJSON from a web resouce, doing a bit of data massaging and pushing out to CartoDB with the CartoDB writer (You'll need to add your own CartoDB api key and a/c into the writer settings).

Just as an additional note....there's no option to truncate the CartoDB in the CartoDB writer format parameters. It would be extremely useful to have this option as we currently have to use a work around using a sampler and httpcaller transformer to do this. Not a big deal but it could be slicker just having this built into the writer (just like it is with the Google Sheets writer)ampersand-test-workspace.fmw

@Mark2AtSafe Hi Mark and thanks for the reply. I've still got the same problem with fme fudging the ampersands. I'm using a work around at the moment and writing the datasets out to Google Sheets which are synced with CartoDB tables. That's working fine.

Sterling Geo, our FME suppliers, are monitoring this thread so they can take up with Safe direct if need be. In the meantime I've attached the workspace I'm using. It's pulling in GeoJSON from a web resouce, doing a bit of data massaging and pushing out to CartoDB with the CartoDB writer (You'll need to add your own CartoDB api key and a/c into the writer settings).

Just as an additional note....there's no option to truncate the CartoDB in the CartoDB writer format parameters. It would be extremely useful to have this option as we currently have to use a work around using a sampler and httpcaller transformer to do this. Not a big deal but it could be slicker just having this built into the writer (just like it is with the Google Sheets writer)ampersand-test-workspace.fmw

Thanks. I passed the workspace on and we'll see what happens.

Yes, the truncate option is a known issue. We're working on adding that (fyi it's PR#60754) and I hope to see it soon. I'll add your details so you get informed when it is. Cheers.

Hi @mark2atsafe,@james_rutter,

I ran into probably into the same trouble.

The mangling is caused by the special meaning that the ampersand has for the operating system.

A workaround is to generate a text that has no meaning anymore, say an ampersand followed by a space and replace that superfluous space with no replacementtext.

Like in the accompanying .fmwt which contains an Excelsheet that reproduces the error.

ampersand.fmwt

Reply


Community Stats

30,177
Posts
114,549
Replies
37,838
Members
Terms & ConditionsCookie settings
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.