Skip to main content
Solved

How to extract URLs from HTML and create features?


Forum|alt.badge.img

Hi FME ninjas,

I'm using the HTTPCaller to call a website containing multiple URLs in its HTML.

 

I'm trying to extract all the URLs that have below format, based on these I'd like to create and attribute (URL) that shows all unique urls as features.

 

What I have:

 

What I want:

 

 

Does this make any sense ;-) ?

 

 

Thanks,

Eduard

Best answer by nielsgerrits

**Update**

With the addition of @takashi, something like this might be what you need.

htmlextractor2018.fmwt

**Original**

Looks like this question?

HtmlExtractor, a[href] as CSS Selector. Does this work in your situation? Did not try yet...

View original
Did this help you find an answer to your question?

10 replies

nielsgerrits
VIP
Forum|alt.badge.img+54
  • Best Answer
  • March 12, 2019

**Update**

With the addition of @takashi, something like this might be what you need.

htmlextractor2018.fmwt

**Original**

Looks like this question?

HtmlExtractor, a[href] as CSS Selector. Does this work in your situation? Did not try yet...


takashi
Influencer
  • March 12, 2019
nielsgerrits wrote:

**Update**

With the addition of @takashi, something like this might be what you need.

htmlextractor2018.fmwt

**Original**

Looks like this question?

HtmlExtractor, a[href] as CSS Selector. Does this work in your situation? Did not try yet...

Yes, the HTMLExtractor does the trick. You can extract the URLs as a list attribute directly with this setting.


Forum|alt.badge.img
  • Author
  • March 12, 2019

Mmm... I can't seem to get any data using above HTMLExtractors.

 

Can you perhaps have a look?

This is the page I'm trying to read: https://www.primera.nl/winkels/

My workbench so far:

 

 

Cheers,

 

Ed

 


takashi
Influencer
  • March 12, 2019
edhere wrote:

Mmm... I can't seem to get any data using above HTMLExtractors.

 

Can you perhaps have a look?

This is the page I'm trying to read: https://www.primera.nl/winkels/

My workbench so far:

 

 

Cheers,

 

Ed

 

Yes I can.


Forum|alt.badge.img
  • Author
  • March 12, 2019
takashi wrote:

Yes I can.

Oh oh... I forgot to explode the list huh?


nielsgerrits
VIP
Forum|alt.badge.img+54
takashi wrote:

Yes, the HTMLExtractor does the trick. You can extract the URLs as a list attribute directly with this setting.

Learned something new today, thanks.


Forum|alt.badge.img
  • Author
  • March 12, 2019

Thank you @nielsgerrits / @takashi for your quick responses.

 

I have exploded the URL list and I now have the data I was looking for.

 

Best,

 

Ed

jkr_wrk
Influencer
Forum|alt.badge.img+29
  • March 12, 2019
edhere wrote:

Oh oh... I forgot to explode the list huh?

 

With the following settings I get a list of 511 shops.

Indeed with a ListExploder. I always forget it is called that.

 


Forum|alt.badge.img
  • Author
  • March 12, 2019
jkr_da wrote:

 

With the following settings I get a list of 511 shops.

Indeed with a ListExploder. I always forget it is called that.

 

I currently have 493.

 

I will try your approach, thanks for sharing.

takashi
Influencer
  • March 12, 2019
edhere wrote:

Mmm... I can't seem to get any data using above HTMLExtractors.

 

Can you perhaps have a look?

This is the page I'm trying to read: https://www.primera.nl/winkels/

My workbench so far:

0684Q00000ArL8HQAV.png

 

 

Cheers,

 

Ed

 

Some pattern matching are available in CSS selector expression. For example, this setting extracts only URLs which begin with "https". See here to learn more: CSS Selector Reference

a[href^="https"]

In addition, you can also download the source HTML directly with the HTMLExtractor.

0684Q00000ArMbuQAF.png


Reply


Cookie policy

We use cookies to enhance and personalize your experience. If you accept you agree to our full cookie policy. Learn more about our cookies.

 
Cookie settings