Skip to main content
  • EN

Hey there,

I have to download several CSV Files from a website. I do it with the HTTPCALLER Transformer. It works perfect. I get ALL the data (several columns and lines) in one "cell":

imageI need the informations in lines and columns, so that I can manipulate the data.

I have tried with Pythoncaller, ListExploder, AttributeCreator,... but it doesn't work.

Do you have any idea, how can I get that? I think the problem is the configuration of the HTTPCALLER or of the other Transformers but I have tried a lot of things and nothing works.

 

Many thanks in advance!

 

Multiple ways:

  • Writing and reading as a file. Will cost some I/O, but is probably the fastest. You can write to temp files to avoid having temp files to clear afterwards using the TempPathnameCreator. This is the way I prefer.
    • Write as CSV using a FeatureWriter.
    • Write as CSV from HTTPCaller.
    • Read written file using a FeatureReader.
  • Splitting the attributes to a list. Create attributes from listelements.
    • Split file to lines based on newline.
    • Split lines to columns based on semicolon.
    • Create attributes from listelements.

Multiple ways:

  • Writing and reading as a file. Will cost some I/O, but is probably the fastest. You can write to temp files to avoid having temp files to clear afterwards using the TempPathnameCreator. This is the way I prefer.
    • Write as CSV using a FeatureWriter.
    • Write as CSV from HTTPCaller.
    • Read written file using a FeatureReader.
  • Splitting the attributes to a list. Create attributes from listelements.
    • Split file to lines based on newline.
    • Split lines to columns based on semicolon.
    • Create attributes from listelements.

No need for the FeatureWriter, you can choose save response body to file instead of attribute in the http caller

Multiple ways:

  • Writing and reading as a file. Will cost some I/O, but is probably the fastest. You can write to temp files to avoid having temp files to clear afterwards using the TempPathnameCreator. This is the way I prefer.
    • Write as CSV using a FeatureWriter.
    • Write as CSV from HTTPCaller.
    • Read written file using a FeatureReader.
  • Splitting the attributes to a list. Create attributes from listelements.
    • Split file to lines based on newline.
    • Split lines to columns based on semicolon.
    • Create attributes from listelements.

Attached workspace demonstrating this.

 

Attached workspace demonstrating this.

 

Hello,

thanks for the answer. If I save the URL as CSV with the featurewriter and after that I read it as featureReader, it goes so fast, that the informations won't be as attribute read... I suppouse that the reader is too fast and the file was not saved

Multiple ways:

  • Writing and reading as a file. Will cost some I/O, but is probably the fastest. You can write to temp files to avoid having temp files to clear afterwards using the TempPathnameCreator. This is the way I prefer.
    • Write as CSV using a FeatureWriter.
    • Write as CSV from HTTPCaller.
    • Read written file using a FeatureReader.
  • Splitting the attributes to a list. Create attributes from listelements.
    • Split file to lines based on newline.
    • Split lines to columns based on semicolon.
    • Create attributes from listelements.

The second way works perfect. Awesome! thanks

Attached workspace demonstrating this.

 

You need to expose the attributes manually, or using import.

The second way works perfect. Awesome! thanks

Cheers :)

Reply


Community Stats

30,113
Posts
114,305
Replies
37,743
Members
Terms & ConditionsCookie settings
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.