Skip to main content
Solved

How to calculate the time for NetCDF-files?

U
Forum|alt.badge.img

Dear FME Users,

 

 

I’d like to process 73 input NetCDF-files. The files describe 3 days (73 hours). Every single file stands for one hour. The NetCDF-files don’t include any useful time attribute.

 

 

What I need is the timestamp for every single input written in an attribute .The data type should be “date”.

 

 

In the following you can take a look to the assignments (input files -> time):

 

- HRES_ENS_2017062200+019 -> June 22, 2017 “07:00 p.m.”

 

- HRES_ENS_2017062200+025 -> June 23, 2017 “07:00 a.m.”

 

- HRES_ENS_2017062200+071 -> June 24, 2017 “11:00 p.m.”

 

 

What I can do in FME is to extract the substrings I need (e.g. “2017-06-22-19”).

 

 

I’ve tried following transformers too: “DateFormatter”, “DateTimeCalculator” and “TimeStamper”.

 

 

Do you’ve an idea how to solve it?

 

 

Thank you very much and best regards!

 

 

Konrad

Best answer by uba_kp

Dear FME-Users;

 

 

I’ve asked you to help me to implement the moving average into my process. I’ve done this (https://knowledge.safe.com/questions/41829/calculate-the-median-of-overlapping-pixel-values-f.html?childToView=50508#comment-50508).

 

 

Now the process can’t calculate the time anymore because the RasterBandCombiner destroys the information I need from the attribute fme_basename.

 

 

I’ve added two processing-screenshots.

 

 

Do you’ve an idea what I can do?

 

 

Thank you very much and best regards!

 

 

Konrad
View original
Did this help you find an answer to your question?

23 replies

danilo_fme
Evangelist
Forum|alt.badge.img+41
  • danilo_fmeEvangelist
  • Evangelist
  • July 26, 2017

Hi @UBA_KP,

I created the Workspace file and the result is:

Please see attachment.

Thanks,

Danilo-workspace-string.fmw

U
Forum|alt.badge.img
  • uba_kp
  • Author
  • July 26, 2017

Thank you very much @danilo_inovacao. The attribut "_result" has the data type "date"?

How can I solve this problem? --> HRES_ENS_2017062200+071 -> June 24, 2017 “11:00 p.m.”

danilo_fme
Evangelist
Forum|alt.badge.img+41
  • danilo_fmeEvangelist
  • Evangelist
  • July 26, 2017
uba_kp wrote:

Thank you very much @danilo_inovacao. The attribut "_result" has the data type "date"?

How can I solve this problem? --> HRES_ENS_2017062200+071 -> June 24, 2017 “11:00 p.m.”

Hello @UBA_KP,

 

 

About your request, would you like to have this format ( 2017-06-22-19 ) in which attribute:

 

 

this HRES_ENS_2017062200+071? or this June 24, 2017 “11:00 p.m.”?

 

 

Thanks

 

 

 

U
Forum|alt.badge.img
  • uba_kp
  • Author
  • July 26, 2017
danilo_fme wrote:
Hello @UBA_KP,

 

 

About your request, would you like to have this format ( 2017-06-22-19 ) in which attribute:

 

 

this HRES_ENS_2017062200+071? or this June 24, 2017 “11:00 p.m.”?

 

 

Thanks

 

 

 

I've this: HRES_ENS_2017062200+071

 

And I'd like to have this: June 24, 2017 “11:00 p.m

 

 

Thank you very much!

 

danilo_fme
Evangelist
Forum|alt.badge.img+41
  • danilo_fmeEvangelist
  • Evangelist
  • July 26, 2017
uba_kp wrote:
I've this: HRES_ENS_2017062200+071

 

And I'd like to have this: June 24, 2017 “11:00 p.m

 

 

Thank you very much!

 

Perfect.

 

How can i get the number 24 in ( 2017062200+071 ) What is the rule?

 

 

thanks

 

 

takashi
Contributor
Forum|alt.badge.img+19
  • takashiContributor
  • Contributor
  • July 26, 2017

Hi @UBA_KP, if I understand your requirement correctly, this procedure might help you.

1. StringSearcher - Extract YYYYmmddHH and hours that should be added.

  • Contain Regular Expression: (\\d+)\\+(\\d+)
  • Subexpression Matches List Name: _list
2. DateTimeCalculator - Get desired datetime in the format YYYYmmddHHMMSS.
  • Mode: Add or Subtract Interval
  • Datetime: @Value(_list{0}.part)0000
  • Operator: Add
  • Hours: _list{1}.part
3. DateFormatter - Format the datetime string.
  • Source Date Format: %Y%m%d%H%M%S (or "Unknown - Automatic Detection")
  • Destination Date Format: %B %d, %Y \\"%I:%M %p\\"
4. StringPairReplacer: Replace "AM" and "PM" with "a.m." and "p.m."
  • Replacement Pairs: AM a.m. PM p.m.

takashi
Contributor
Forum|alt.badge.img+19
  • takashiContributor
  • Contributor
  • July 26, 2017
Why should 00+025 hours be 07:00 a.m.?

 

HRES_ENS_2017062200+025 -> June 23, 2017 “07:00 a.m.”

 

U
Forum|alt.badge.img
  • uba_kp
  • Author
  • July 26, 2017
takashi wrote:
Why should 00+025 hours be 07:00 a.m.?

 

HRES_ENS_2017062200+025 -> June 23, 2017 “07:00 a.m.”

 

Sorry it's a mistake, it should be "01:00 a.m".

 

 

U
Forum|alt.badge.img
  • uba_kp
  • Author
  • July 26, 2017
takashi wrote:

Hi @UBA_KP, if I understand your requirement correctly, this procedure might help you.

1. StringSearcher - Extract YYYYmmddHH and hours that should be added.

  • Contain Regular Expression: (\\d+)\\+(\\d+)
  • Subexpression Matches List Name: _list
2. DateTimeCalculator - Get desired datetime in the format YYYYmmddHHMMSS.
  • Mode: Add or Subtract Interval
  • Datetime: @Value(_list{0}.part)0000
  • Operator: Add
  • Hours: _list{1}.part
3. DateFormatter - Format the datetime string.
  • Source Date Format: %Y%m%d%H%M%S (or "Unknown - Automatic Detection")
  • Destination Date Format: %B %d, %Y \\"%I:%M %p\\"
4. StringPairReplacer: Replace "AM" and "PM" with "a.m." and "p.m."
  • Replacement Pairs: AM a.m. PM p.m.

Thank you very much @takashi!! Maybe it's possible for you to make some screenshots? It's much easier for me to understand. Thank you very much! :)

 

takashi
Contributor
Forum|alt.badge.img+19
  • takashiContributor
  • Contributor
  • July 26, 2017
takashi wrote:

Hi @UBA_KP, if I understand your requirement correctly, this procedure might help you.

1. StringSearcher - Extract YYYYmmddHH and hours that should be added.

  • Contain Regular Expression: (\\d+)\\+(\\d+)
  • Subexpression Matches List Name: _list
2. DateTimeCalculator - Get desired datetime in the format YYYYmmddHHMMSS.
  • Mode: Add or Subtract Interval
  • Datetime: @Value(_list{0}.part)0000
  • Operator: Add
  • Hours: _list{1}.part
3. DateFormatter - Format the datetime string.
  • Source Date Format: %Y%m%d%H%M%S (or "Unknown - Automatic Detection")
  • Destination Date Format: %B %d, %Y \\"%I:%M %p\\"
4. StringPairReplacer: Replace "AM" and "PM" with "a.m." and "p.m."
  • Replacement Pairs: AM a.m. PM p.m.

Hope this helps.

 

 

U
Forum|alt.badge.img
  • uba_kp
  • Author
  • July 26, 2017

Thank you very much @takashi. It seems that I've set some parameters wrong in the transformer "DateTimeCalculator". Take a look in the attachment please. Maybe you'll find the mistake. Thank you again!

takashi
Contributor
Forum|alt.badge.img+19
  • takashiContributor
  • Contributor
  • July 26, 2017
uba_kp wrote:

Thank you very much @takashi. It seems that I've set some parameters wrong in the transformer "DateTimeCalculator". Take a look in the attachment please. Maybe you'll find the mistake. Thank you again!

oops , probably you are using FME 2016.1. The DateTimeCalculator shown in your screenshot is a custom transformer from FME Hub. It has been deprecated in 2017, and the same name regular transformer has been newly introduced instead.

 

In FME 2016.1, try this setting.

 

 

 

U
Forum|alt.badge.img
  • uba_kp
  • Author
  • July 26, 2017

Now it seems to work, thank you very much @takashi.

But the date is not correct, please take a look in the attachment.

The file I've chosen is "HRES_ENS_2017062200+019". The result must be "June 22, 2017 “07:00 p.m.”".

When I use the data typ "date" instead of "text" in the feature writer, all fields are empty.

What can I do?

takashi
Contributor
Forum|alt.badge.img+19
  • takashiContributor
  • Contributor
  • July 26, 2017
uba_kp wrote:

Now it seems to work, thank you very much @takashi.

But the date is not correct, please take a look in the attachment.

The file I've chosen is "HRES_ENS_2017062200+019". The result must be "June 22, 2017 “07:00 p.m.”".

When I use the data typ "date" instead of "text" in the feature writer, all fields are empty.

What can I do?

If you need to write a date/time value into a date/time type field of a destination feature type, you will have to format the date string with the Standard FME Date/Time Format (typically %Y%m%d%H%M%S) in general.

 

U
Forum|alt.badge.img
  • uba_kp
  • Author
  • July 28, 2017
takashi wrote:
oops , probably you are using FME 2016.1. The DateTimeCalculator shown in your screenshot is a custom transformer from FME Hub. It has been deprecated in 2017, and the same name regular transformer has been newly introduced instead.

 

In FME 2016.1, try this setting.

 

 

 

Now it works, thank you so much @takashi.

 

 

Best regards.

 

 

Konrad
U
Forum|alt.badge.img
  • uba_kp
  • Author
  • Best Answer
  • August 15, 2017

Dear FME-Users;

 

 

I’ve asked you to help me to implement the moving average into my process. I’ve done this (https://knowledge.safe.com/questions/41829/calculate-the-median-of-overlapping-pixel-values-f.html?childToView=50508#comment-50508).

 

 

Now the process can’t calculate the time anymore because the RasterBandCombiner destroys the information I need from the attribute fme_basename.

 

 

I’ve added two processing-screenshots.

 

 

Do you’ve an idea what I can do?

 

 

Thank you very much and best regards!

 

 

Konrad
U
Forum|alt.badge.img
  • uba_kp
  • Author
  • August 15, 2017
uba_kp wrote:

Dear FME-Users;

 

 

I’ve asked you to help me to implement the moving average into my process. I’ve done this (https://knowledge.safe.com/questions/41829/calculate-the-median-of-overlapping-pixel-values-f.html?childToView=50508#comment-50508).

 

 

Now the process can’t calculate the time anymore because the RasterBandCombiner destroys the information I need from the attribute fme_basename.

 

 

I’ve added two processing-screenshots.

 

 

Do you’ve an idea what I can do?

 

 

Thank you very much and best regards!

 

 

Konrad
@takashi maybe you've an idea how to solve it? Thank your very much! :)

 

takashi
Contributor
Forum|alt.badge.img+19
  • takashiContributor
  • Contributor
  • August 15, 2017
uba_kp wrote:

Dear FME-Users;

 

 

I’ve asked you to help me to implement the moving average into my process. I’ve done this (https://knowledge.safe.com/questions/41829/calculate-the-median-of-overlapping-pixel-values-f.html?childToView=50508#comment-50508).

 

 

Now the process can’t calculate the time anymore because the RasterBandCombiner destroys the information I need from the attribute fme_basename.

 

 

I’ve added two processing-screenshots.

 

 

Do you’ve an idea what I can do?

 

 

Thank you very much and best regards!

 

 

Konrad
There are several numbers regarding the number of rasters or bands, so I'm unclear what the source dataset is and what the destination dataset is

 

The source are 73 single band rasters for each date and the destination are 50 single band rasters for each moving average. Is it right?

 

 

U
Forum|alt.badge.img
  • uba_kp
  • Author
  • August 15, 2017
takashi wrote:
There are several numbers regarding the number of rasters or bands, so I'm unclear what the source dataset is and what the destination dataset is

 

The source are 73 single band rasters for each date and the destination are 50 single band rasters for each moving average. Is it right?

 

 

I've 97 input rasters and every single raster has 1 band. After processing the moving average I've 73 rasters.

 

takashi
Contributor
Forum|alt.badge.img+19
  • takashiContributor
  • Contributor
  • August 15, 2017
uba_kp wrote:

Dear FME-Users;

 

 

I’ve asked you to help me to implement the moving average into my process. I’ve done this (https://knowledge.safe.com/questions/41829/calculate-the-median-of-overlapping-pixel-values-f.html?childToView=50508#comment-50508).

 

 

Now the process can’t calculate the time anymore because the RasterBandCombiner destroys the information I need from the attribute fme_basename.

 

 

I’ve added two processing-screenshots.

 

 

Do you’ve an idea what I can do?

 

 

Thank you very much and best regards!

 

 

Konrad
The workflow I suggested to create moving average rasters keeps the order of the original rasters and the order (0-based sequential number) is stored in "_copynum" attribute. I therefore think that you can merge "fme_basename" of the original raster to the resulting moving average raster using an attribute representing the input order as join key. e.g.

 

 

U
Forum|alt.badge.img
  • uba_kp
  • Author
  • August 15, 2017
takashi wrote:
The workflow I suggested to create moving average rasters keeps the order of the original rasters and the order (0-based sequential number) is stored in "_copynum" attribute. I therefore think that you can merge "fme_basename" of the original raster to the resulting moving average raster using an attribute representing the input order as join key. e.g.

 

 

@takashi thank you very much, but I think that I don't need the cloner. I already have 97 input raster. Do you've an idea which transformer I can use instead of cloner?

 

takashi
Contributor
Forum|alt.badge.img+19
  • takashiContributor
  • Contributor
  • August 16, 2017
takashi wrote:
The workflow I suggested to create moving average rasters keeps the order of the original rasters and the order (0-based sequential number) is stored in "_copynum" attribute. I therefore think that you can merge "fme_basename" of the original raster to the resulting moving average raster using an attribute representing the input order as join key. e.g.

 

 

The workflow I suggested in this link requires to use the Cloner to create groups of single band rasters for each range of moving average.

 

Calculate the median of overlapping pixel values from 24 input raster bands

 

I don't know the actual workflow you have adopted, since there is no detailed explanation on your final solution. However, I believe that there should be a way to merge attributes including "fme_basename" of the source rasters to the moving average rasters according to the original input order, if your workflow keeps the input order. The specific way depends on the implementation of your workflow to create the moving average rasters.

 

U
Forum|alt.badge.img
  • uba_kp
  • Author
  • December 12, 2017
takashi wrote:
The workflow I suggested in this link requires to use the Cloner to create groups of single band rasters for each range of moving average.

 

Calculate the median of overlapping pixel values from 24 input raster bands

 

I don't know the actual workflow you have adopted, since there is no detailed explanation on your final solution. However, I believe that there should be a way to merge attributes including "fme_basename" of the source rasters to the moving average rasters according to the original input order, if your workflow keeps the input order. The specific way depends on the implementation of your workflow to create the moving average rasters.

 

Thank you very much @takashi. Now it works! I’ve used number “-24” as “Count Start” for the Transformer “Counter”. I already wrote why my answer is so late (holiday e.g.).

 

Reply


Helpful Members This Week

Community Stats

30,918
Posts
117,146
Replies
38,721
Members

Latest FME

Terms & ConditionsCookie settings

Cookie policy

We use cookies to enhance and personalize your experience. If you accept you agree to our full cookie policy. Learn more about our cookies.

 
Cookie settings
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.