Question

Joiner doesn't handle images stored in a blob ?

  • 24 August 2017
  • 5 replies
  • 2 views
T

Hello,

We have a database (SQL-server) where we have stored water pipes with connected jpg-pictures (1 pipe:many pictures). The data is collected with "Collector for ArcGIS" and the pipes are stored in one table and each pipe has its unique guid. The Pictures are stored in a separate table and the Connection between the pipes and the pictures is handled by a "REL_GUID" attribute.

The thing I have to do is to extract the pictures from the database and store them as "normal files" on a disc and write the URL for the pictures to an attribute on the waterpipe object.

Just before the summer vacation I wrote a FME-script that read the two tables with an ArcSDE Geodb-reader. I then joined the water pipes and the pictures with FeatureMerger and the result was a water pipe object with a list containing all relevant information from the picture table, including the actual data blob. The next step was to write the picture to a relevant disc using AttributeFileWriter. Everything worked fine while testing so I went on vacation. But the field staff didn’t and they were busy filling the database and now due to the amount of data FeatureMerger dies the out-of-memory-death.

I tried to use the Joiner transformer instead but now the pictures get corrupted when written to disc. With my limited knowledge it seems like there is a bug in the Joiner. Any ideas for an alternative solution anyone ?

I’m running the script on a 32-BIT FME 2016.1

Regards

Thomas Norlin

5 replies

M
Badge +2

Are you using the Suppliers First = Yes option on the FeatureMerger? You have guarantee that all the Supplier features arrive at the transformer before any Requestor features. You can do this using some tricks such as two Readers connecting to the same database and using the order in the Navigator to control which is read first (assumes you are reading with standard Reader rather than FeatureReader).

However the image BLOB could be pushing up the memory also. There could be another approach where you make the join without the BLOB, then refetch the BLOB after the join.

Uploading the Workspace would help.

T

Tanks,

I tried the Supplier First option but the result was still "out of memory". Then I tried to upgrade to 2017.1 and to use the transformer DatabaseJoiner in the same way as I used Joiner but with limited success. Finally I upgraded to 2017.1 64 BIT with FeatureMerger and then everyting worked fine.

Regards

Thomas

M
Badge +2

Tanks,

I tried the Supplier First option but the result was still "out of memory". Then I tried to upgrade to 2017.1 and to use the transformer DatabaseJoiner in the same way as I used Joiner but with limited success. Finally I upgraded to 2017.1 64 BIT with FeatureMerger and then everyting worked fine.

Regards

Thomas

 

Great! Was going to suggest 64 bit but thought you may have been restricto 32 bit.
T

collectorpipes-safe17.fmw

Here's the script

T

Another solution (in case somebody else reads this) is to read the table with an inner join in SQLCreator. But I'm afraid that I still will have problems in the future. FME 2017 64 Bit is the best solution

Reply


Community Stats

28,668
Posts
109,356
Replies
36,532
Members
Terms & ConditionsCookie settings
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.