FME Server - Published parameter as utf-8 or unicode?

Hi @sigtill

I did a little research and it may be a browser problem. ie the browser should encode those characters in a URL style when it sends the request to FME Server. Have you tried a different browser? It might not help as a solution, but it would perhaps show whether the problem is browser-specific.

Another clue would be in the URL provided in the 'developer info' part of the FME Server web interface. Is the information correctly encoded there (as %C3%A6)? Or is it incorrect there too? If it's incorrect there then I definitely think it's an FME issue.

Does that help? If not I suggest you contact our support team and ask for assistance. They should be able to tell you where the source of the problem is.

Regards

Mark

Hi Sigbjørn

UTF-8 is actually just a way of representing Unicode.

What you've stumbled upon is what's called URL Encoding, or Percent Encoding, which is pretty well explained in the wikipedia article.

If you need to convert to/from URL Encoding inside a workspace, look at the TextEncoder transformer.

In Javascript you can use the built-in functions encodeURIComponent() and decodeURIComponent() to accomplish the same thing.

David

Mark2AtSafe:

The situation described by Sigtill is a bit more complex.

We are calling the datadownload through C#-code server-side, relevant code here: https://gist.github.com/atlefren/722660e2112d6660b...

As far as I understand everything related to charset is OK before the actual POST request is sent to FME server, but in the generated pdf our service creates the characters æøå are simply dropped, while standard ascii characters are printed as expected.

Yet stranger, when looking at the status page for the FME server, the request sent from our C#-code actually has æøå, and they are printed fine on the webpage, as FME_SERVER_REQUEST_PARAMETERS

{"opt_responseformat":"json","opt_requesteremail":"....","Buffer":"20","Overskrift":"Test æøå!","opt_servicemode":"async","geomPolygon":"....."}

Hi @a4, in my experiences, there are cases that the PDF writer cannot write non-ascii characters. If you run the workspace with Workbench, those characters will be written into a pdf doc correctly?

Hi takashi - PDF is written OK when run from FME Desktop and FME Servers web-interface (www.myfmeserverurl.com/fmeserver) - so the error is as mentioned by a4 when calling the workspace through c#-code serverside.

hmm, I'm not sure the exact reason, but guess that FME Server interprets the encoded parameters with the default encoding of its platform, which may not be UTF-8.

It might be worth to try specifying the encoding explicitly to the second argument of the "System.Net.WebUtility.UrlEncode" method in your C# code.

HttpUtility.UrlEncode Method (String,?Encoding)

I'm having a similar issue when calling a published parameter within a python scripted parameter in FME Desktop. Lets say I have a return line in my script like this: return 'æøå', this works fine, but when I try to return a published parameter containing 'æøå' like this: return FME_MacroValues['My_Param'], FME 2017.0.2 stops, no crash report or nothing. When printing the two values, print 'æøå' shows æøå, but print FME_MacroValues['My_Param'] shows. <?><?><?>. If I write both of them to a text file, like this txt= 'C:\\\\temp\\\\test.txt' with open(txt,'w') as t: t.write('æøå') t.write(FME_MacroValues['My_Param']), In both cases it writes æøå. I have tried to encode and decode FME_MacroValues['My_Param'].encode('UFT-8') and iso-8859-1 with no luck. I think there might be something with the coding of published parameters, but I am out of ways to solve this now. Anyone knows how to get pass this?

My suggestion to improve the Published text parameter

def ParamFunc():
  return u'æøå'

value = ParamFunc()
macroName = 'my_favorite_letters'
if value == None:
  return { macroName : '' }
else:
  return { macroName : str(value) }

I've posted the following idea which you can vote for:

https://knowledge.safe.com/idea/44684/python-scripted-parameters-must-be-able-to-return.html

This would fix the encoding issues when returning a unicode string in a Python scripted parameter.

 I found an odd solution to my own question:

import base64
letters = FME_MacroValues['My_Favourite_Letters']
return base64.b64encode(letters)

This actually solves the whole problem. I thought I had to convert this back to ascii with the BinaryEncoder Transformer with base64, but I didn't need to. It was allready encoded correctly.