HTTPCaller version 2022.2

I think you'll have to show us example data, or perhaps even better, post a sample workspace that demonstrates the issue.

Secondly, "wwjd", someone at development clearly had an epiphany that day ;-)

Hi @david_r​ ,

this is the configuration of HTTPCaller

And I attach JSON file

By the way, I added AttributeEncoder transformer, before HttpCaller transformer, in order to change encoding, but it didn't work.

Hi @david_r​ 

thanks for your help. I've just tried what you propose but, unfortunately, it didn't work.

Thank you so much @david_r​, it works!!!

I'll try to reach out with our FME reseller to confirm this point.

I also experience problems with the HTTPCaller 2022.2 and would appreciate a statement from SAFE regarding this problem - and if there are any workarounds.

Scripts I could use in 2021 don't work anymore. Specifically I'm having a hard time updating JSON-chunks for items in ArcGIS Enterprise via the REST API...

Hi @mark2atsafe​ and @Mark Stoakes​  do you know if this is a known issue?

Hi @dxirgu​,

This problem seems to have introduced between FME 2022.1 and FME 2022.2. I have created a problem report for our development team (FMEENGINE-75726), and expect to see a resolution in FME 2023.

I will notify you if we find a workaround, but you may want to replace your FME 2022.2 installation with FME 2022.1.

@daveatsafe​ We update all our desktop installations at the same time, as well as FME Server and Cloud so downgrading is not an option. No sightings of a workaround?

Thank you so much @daveatsafe​ !!!

Hi @dxirgu​, @david_r​, and @peteralstorp​,

I'm pleased to share that the problem report Dave filed above, (FMEENGINE-75726), has been resolved with a fix for FME 2022.2.2 and newer. You can find the fix in both the official and beta builds on our downloads page. Please let us know if you should have any feedback on it!

Hey @jovitaatsafe​ . In what build number was this resolved? I have this exact problem on our FME Server, 2022.2 Build 22765 Win64 . Are there any known workarounds?

According to https://engage.safe.com/support/downloads/#past-versions, FME 2022.2.2 corresponds to build number 22782. There's a suggested workaround mentioned at the top of this post, i.e. removing the diacritics or replacing the problematic characters.

Hi @krisvesweco​, the minimum build on the fix is build 22778, so it looks like your version is just below that if you'd like to upgrade or use ​@david_r​'s workaround of using the StringDiacriticRemover before the HTTPCaller.

Hi @krisvesweco​, it looks like this issue was resolved as of build 22778. Would you be able to download and test out a newer build?

Thank you for you input, @david_r​ , @jovitaatsafe​  and @saraatsafe​ 

Well, upgrading the FME server will take some time and planning unfortunately.

 I was hoping for a workaround that could be used that still makes us use the Swedish characters . If I cant set certain words with Swedish characters I cant upload to the service at all since it is required fields. So if I remove the diacritics the service will reject my json.

It brings me no joy to say this, but if you're completely stumped then it might be time to break out Python and the trusty "requests" module.

Great, good idea @david_r​ ! I successfully built a workaround using requests and pythoncaller. I post some code here to help others that might experience this problem:

import fme
import fmeobjects
from pathlib import Path
import json
from fmewebservices import FMENamedConnectionManager
import requests
from requests.packages.urllib3.util.retry import Retry
from requests.adapters import HTTPAdapter
 
def requests_retry_session(
    retries=3,
    backoff_factor=0.3,
    status_forcelist=(500, 502, 504),
    session=None,):
    session = session or requests.Session()
    retry = Retry(
        total=retries,
        read=retries,
        connect=retries,
        backoff_factor=backoff_factor,
        status_forcelist=status_forcelist,
    )
    adapter = HTTPAdapter(max_retries=retry)
    session.mount('http://', adapter)
    session.mount('https://', adapter)
    return session
 
 
class FeatureProcessor(object):
    def __init__(self):
        pass
 
    def input(self, feature):
        connection_manager = FMENamedConnectionManager()
        connection_name = 'your_connection_name_to_use'
        connection_info = connection_manager.getNamedConnection(connection_name)
        
        headerToken = connection_info.getAuthorizationHeader()
        #fix tuple to dict
        headerToken = [headerToken]
        headerToken = dict(headerToken)
        headerToken['Authorization'] = headerToken.pop('Authorization: ')
 
        
        file_to_upload = feature.getAttribute('path_windows')
        url = 'https://your_url_to_rest_api'
        
        #create a dictionary
        data = {}
        data['name'] = feature.getAttribute('path_rootname')
        data['documentType'] = {}
        data['documentType']['name'] = feature.getAttribute('documentType')
        data['customFields'] = {}
        data['customFields']['Status'] = {}
        data['customFields']['Status']['value'] = feature.getAttribute('Status')
        data['parent'] = {}
        data['parent']['type'] = "folder"
        data['parent']['id'] = feature.getAttribute('folderID')        
 
        encoded_data = json.dumps(data, ensure_ascii=False).encode('utf8')
        
        files = {
            'data': (None, encoded_data, 'application/json'),
            'file': (feature.getAttribute('WF_file_rootname') + '.' + feature.getAttribute('WF_file_extension'), open(file_to_upload, 'rb'),'application/octet-stream')
        }
 
        successfulresponse = False
        try:
            WFresponse = requests_retry_session().post(url, files=files, headers=headerToken, timeout=10)
            
        except Exception as x:
            _http_status_code = 'Connection timed out, restarting application... ' + x.__class__.__name__ + ' ' + 'uploadNewFileWF'
        else:#code that must be executed if the try clause does not raise an exception
            successfulresponse = True
            _http_status_code = WFresponse.status_code
            
        feature.setAttribute('_http_status_code', _http_status_code)
 
        self.pyoutput(feature)
 
    def close(self):
        pass
 
    def process_group(self):
        pass
 
    def has_support_for(self, support_type):
        if support_type == fmeobjects.FME_SUPPORT_FEATURE_TABLE_SHIM:
            return False
 
        return False

import fme
import fmeobjects
from pathlib import Path
import json
from fmewebservices import FMENamedConnectionManager
import requests
from requests.packages.urllib3.util.retry import Retry
from requests.adapters import HTTPAdapter
 
def requests_retry_session(
    retries=3,
    backoff_factor=0.3,
    status_forcelist=(500, 502, 504),
    session=None,):
    session = session or requests.Session()
    retry = Retry(
        total=retries,
        read=retries,
        connect=retries,
        backoff_factor=backoff_factor,
        status_forcelist=status_forcelist,
    )
    adapter = HTTPAdapter(max_retries=retry)
    session.mount('http://', adapter)
    session.mount('https://', adapter)
    return session
 
 
class FeatureProcessor(object):
    def __init__(self):
        pass
 
    def input(self, feature):
        connection_manager = FMENamedConnectionManager()
        connection_name = 'your_connection_name_to_use'
        connection_info = connection_manager.getNamedConnection(connection_name)
        
        headerToken = connection_info.getAuthorizationHeader()
        #fix tuple to dict
        headerToken = [headerToken]
        headerToken = dict(headerToken)
        headerToken['Authorization'] = headerToken.pop('Authorization: ')
 
        
        file_to_upload = feature.getAttribute('path_windows')
        url = 'https://your_url_to_rest_api'
        
        #create a dictionary
        data = {}
        data['name'] = feature.getAttribute('path_rootname')
        data['documentType'] = {}
        data['documentType']['name'] = feature.getAttribute('documentType')
        data['customFields'] = {}
        data['customFields']['Status'] = {}
        data['customFields']['Status']['value'] = feature.getAttribute('Status')
        data['parent'] = {}
        data['parent']['type'] = "folder"
        data['parent']['id'] = feature.getAttribute('folderID')        
 
        encoded_data = json.dumps(data, ensure_ascii=False).encode('utf8')
        
        files = {
            'data': (None, encoded_data, 'application/json'),
            'file': (feature.getAttribute('WF_file_rootname') + '.' + feature.getAttribute('WF_file_extension'), open(file_to_upload, 'rb'),'application/octet-stream')
        }
 
        successfulresponse = False
        try:
            WFresponse = requests_retry_session().post(url, files=files, headers=headerToken, timeout=10)
            
        except Exception as x:
            _http_status_code = 'Connection timed out, restarting application... ' + x.__class__.__name__ + ' ' + 'uploadNewFileWF'
        else:#code that must be executed if the try clause does not raise an exception
            successfulresponse = True
            _http_status_code = WFresponse.status_code
            
        feature.setAttribute('_http_status_code', _http_status_code)
 
        self.pyoutput(feature)
 
    def close(self):
        pass
 
    def process_group(self):
        pass
 
    def has_support_for(self, support_type):
        if support_type == fmeobjects.FME_SUPPORT_FEATURE_TABLE_SHIM:
            return False
 
        return False

Very nice, thanks for sharing!