Skip to main content
Solved

Ignore old files with ProjectWise

S

Hello!

 

I'm reaching out because I'm in aa bit of a pickle. I have a task to pull a large number of photos and folders from a ProjectWise folder. I've successfully been able to do so through the ProjectWiseWSGConnector and FeatureReader, and then a number of other transformers that modifies the images for what we need. However, because the number of photos is quite large (GB's of data), it's quite the task to run it and to re-download all of the files.

 

Is there a workflow that will skip over previously downloaded files and just download the newly added or updated ones?

 

Thanks for any help!

Best answer by andreaatsafe

Hi @spizam,

There are a couple of ways you could do this depending on how you want to set things up.

Are you downloading the photos to a folder? If so, then I would recommend setting up your workflow as follows:

1. Using the Directory and File Path reader, read and filter for the images that have been downloaded already.

2. Use a ProjectWiseWSGConnector, with the "List" action to list all the files from the folder you were downloading from.

3. Then use a FeatureJoiner to join the files based on the photo name - this is assuming that the file names have not changed. I would verify that the attribute, _document_name, from the ProjectWiseWSGConnector holds the same file name as the path_filename attribute

4. Then you can use the unjoined features to feed into the next ProjectWiseWSGConnector to download the files. You can specify the attribute, _object_id, in the ProjectWise Source>Object ID parameter. Then you can continue your workflow as before.

Another way you might want to consider is to use a ProjectWiseWSGConnector with the List action and use a Tester to filter based on the creation date or update date attribute. This would be useful if you are running this on a schedule and only want to download the files created in the last day for example. After the Tester, you can then connect to the remainder of your workflow to download the photos etc.

Hope this helps give you some ideas!

- Andrea

View original
Did this help you find an answer to your question?

4 replies

andreaatsafe
Safer
Forum|alt.badge.img+10
  • andreaatsafeSafer
  • Safer
  • Best Answer
  • July 2, 2020

Hi @spizam,

There are a couple of ways you could do this depending on how you want to set things up.

Are you downloading the photos to a folder? If so, then I would recommend setting up your workflow as follows:

1. Using the Directory and File Path reader, read and filter for the images that have been downloaded already.

2. Use a ProjectWiseWSGConnector, with the "List" action to list all the files from the folder you were downloading from.

3. Then use a FeatureJoiner to join the files based on the photo name - this is assuming that the file names have not changed. I would verify that the attribute, _document_name, from the ProjectWiseWSGConnector holds the same file name as the path_filename attribute

4. Then you can use the unjoined features to feed into the next ProjectWiseWSGConnector to download the files. You can specify the attribute, _object_id, in the ProjectWise Source>Object ID parameter. Then you can continue your workflow as before.

Another way you might want to consider is to use a ProjectWiseWSGConnector with the List action and use a Tester to filter based on the creation date or update date attribute. This would be useful if you are running this on a schedule and only want to download the files created in the last day for example. After the Tester, you can then connect to the remainder of your workflow to download the photos etc.

Hope this helps give you some ideas!

- Andrea

S
  • spizam
  • Author
  • July 14, 2020

Hi @andreaatsafe! This was the perfect solution. Thank you!

I did have one follow up question, and please let me know if this needs to be a separate question submission.

Is there any way to identify the different versions of the same file within PW? At the moment, it seems that all versions are listed as the same name, and while we can download a "List" to find the latest one with an "Update Time", we were curious if there was any other way to identify the differences between the versions.

Thanks so much!

S
  • spizam
  • Author
  • July 15, 2020
spizam wrote:

Hi @andreaatsafe! This was the perfect solution. Thank you!

I did have one follow up question, and please let me know if this needs to be a separate question submission.

Is there any way to identify the different versions of the same file within PW? At the moment, it seems that all versions are listed as the same name, and while we can download a "List" to find the latest one with an "Update Time", we were curious if there was any other way to identify the differences between the versions.

Thanks so much!

Apologies @andreaatsafe - I just thought of an additional follow up question. Based on the workflow you posted above, can subfolder paths be preserved when downloading? For example, if the 1st ProjectWiseWSGConnector pulled a parent folder with a large number of subfolders, and it runs through the FeatureJoiner into the second ProjectWiseWSGConnector, can it download into the appropriate subfolder? Let me know if any of that needs further clarification. Thanks for the help!

andreaatsafe
Safer
Forum|alt.badge.img+10
spizam wrote:

Apologies @andreaatsafe - I just thought of an additional follow up question. Based on the workflow you posted above, can subfolder paths be preserved when downloading? For example, if the 1st ProjectWiseWSGConnector pulled a parent folder with a large number of subfolders, and it runs through the FeatureJoiner into the second ProjectWiseWSGConnector, can it download into the appropriate subfolder? Let me know if any of that needs further clarification. Thanks for the help!

For your first question: With the current transformer, the list action simply pulls information regarding the files from the Project Object ID that is set to list from. So it is not possible for the transformer to look for the latest version of the file that was uploaded to your ProjectWise folder. As mentioned in the second potential workflow, you need to use subsequent transformers in FME to create the logic to do so (eg. Tester to test for the latest date).

 

For your second question:

 

The Download action for the ProjectWiseWSG transformer does allow you to download as a folder, when the "Download as" parameter is set to "Folder". You then have access to the parameter "Include Subfolders", if this is set to "Yes" then it will download sub folders/projects of the ProjectWise Source.

 

So if you initially "List", the folders are listed and classified as follows: 
_class_type = Project
 and 
_object_id

Which would allow you to filter the 

_class_type
 so you could, and connect it to a different ProjectWiseWSGConnector which is set to download a folder. I would highly recommend reviewing the attributes returned when you "List" and then create filters based on if you want to download folders or files.

 

 

I hope this helps provide some more information.

 

- Andrea

Reply


Helpful Members This Week

Community Stats

31,094
Posts
117,893
Replies
38,851
Members

Latest FME

Terms & ConditionsCookie settings

Cookie policy

We use cookies to enhance and personalize your experience. If you accept you agree to our full cookie policy. Learn more about our cookies.

 
Cookie settings
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.