• Home
  • Forums
  • FME Form
  • Data
  • How can I temporarily limit the number of records that I'm reading from an ArcGIS feature layer?
Solved

How can I temporarily limit the number of records that I'm reading from an ArcGIS feature layer?

  • 12 January 2022
  • 8 replies
  • 36 views
datablue
Rank
Badge +7

I'm trying to speed up the processing speed. There might be a few different ways to do this. I'm trying to make incremental changes to a workspace and every time I run it, it takes about 40 seconds.

 

I have two tables in ArcGIS and I use a FeatureJoiner to join them in FME. One of the tables has a data field. I'd like to only process a range of dates. I have about 8,000 records total and I'd like to limit it to 1,000 or so records. Once I get my workspace functioning correctly, I'd like to remove this date range limit. I'm very new to FME.

icon

Best answer by nielsgerrits 12 January 2022, 10:01

View original

8 replies

nielsgerrits
Rank
Userlevel 6
Badge +32

You probably can do this using the where clause in the reader.

What do you use to read the data? A classic reader or a FeatureReader?

Which type of reader? Esri ArcGIS Server Feature Service?

ebygomm
Rank
Userlevel 1
Badge +21

You should be able to do this under the reader parameters in the navigator

image

nielsgerrits
Rank
Userlevel 6
Badge +32

You should be able to do this under the reader parameters in the navigator

image

That was my initial thought as well, but he wants to read a specific date range.

I'd like to only process a range of dates.

datablue
Rank
Badge +7

You probably can do this using the where clause in the reader.

What do you use to read the data? A classic reader or a FeatureReader?

Which type of reader? Esri ArcGIS Server Feature Service?

Thanks for your response. The reader type shows: Esri ArcGIS Online (AGOL) Feature Service. When I double click on it, I see the Where clause in the Parameters tab. It looks straight forward. I'll try to fill it out now. I'm now thinking that I can use the objectid.

 

I have two tables, each with a reader. The first table is for locations and the second table is for people. They have a one-to-many relationship: Locations -> People (each location can have one or more persons).

 

I think that I can specify that objectid needs to be greater than a certain number which could give me the last X number of records. Or, perhaps I can specify a range. I'll give this a try. I suspect that I will need to specify a where clause on the reader for the second table as well.

datablue
Rank
Badge +7

Thanks for your response. The reader type shows: Esri ArcGIS Online (AGOL) Feature Service. When I double click on it, I see the Where clause in the Parameters tab. It looks straight forward. I'll try to fill it out now. I'm now thinking that I can use the objectid.

 

I have two tables, each with a reader. The first table is for locations and the second table is for people. They have a one-to-many relationship: Locations -> People (each location can have one or more persons).

 

I think that I can specify that objectid needs to be greater than a certain number which could give me the last X number of records. Or, perhaps I can specify a range. I'll give this a try. I suspect that I will need to specify a where clause on the reader for the second table as well.

As I mentioned, I have two tables, each with a reader. The first table is for locations and the second table is for people. They have a one-to-many relationship: Locations -> People (each location can have one or more persons).

 

I added a where clause for each reader.

Locations table: "objectid" BETWEEN 23277 AND 23528

People table: "objectid" BETWEEN 4424 AND 4684

 

Now, I'm processing a few hundred records instead of about 8000. The processing time is a few seconds instead of about 40 seconds. Now I can incrementally change my workspace. When I have everything working correctly, I'll remove the Where clause.

 

I couldn't figure out how to filter on the date mostly because only the Locations table had a date and I needed a where clause on both tables/readers. Thanks for everyone's help. This was my first question.

mark2atsafe
Rank
Userlevel 4
Badge +25

You might be new to FME but you're spot on in restricting the input while you develop the workspace. That's a good call.

 

Another way to help would be to use Feature Caching (the Cache button on the toolbar). It takes a bit longer on the first run, but after that, you only need to run the workspace on changes that you make. All existing readers/transformers that don't change have a cache of the data at that stage of the process, which is quicker than re-processing it.

datablue
Rank
Badge +7

Thanks for the tip. I have recently started making use of caching. I how hovering the mouse over the Run button shows the readers/transformer that will update. (I don't see a Cache button on the toolbar, but maybe I'm looking in the wrong place.)

 

For this particular situation, I'm glad that I was able to restrict the input using a where clause in the reader. I had 8,000+ records and limited the input to a few hundred records.

S

This is helpful, thanks for sharing.

Reply


Community Stats

28,710
Posts
109,487
Replies
36,559
Members
Terms & ConditionsCookie settings
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.