Solved

Clip pointcloud when reading

james_c_452
Rank
Badge +4

Does anyone know if it is possible to clip a point cloud to a boundary when reading into a workspace? Often I have the need to only work with part of a point cloud file and my full files are frequently up to 10GB (in las format), though usually smaller. I'm also reading up to 100 similarly sized files at once

 

Basically, I would like to read only the section of a point cloud that's in a boundary to save on time/computing power/memory/etc. If this isn't possible would I be able to thin instead when I read to load less data? I'd like to avoid having to use an external program to clip/thin first if at all possible. I already do this

icon

Best answer by mark2atsafe 17 January 2022, 20:32

View original

16 replies

caracadrian
Rank
Badge +20

Use a search envelope in the reader parameters:

search envelope LASYou can also chose to use a FeatureReader instead of a "classic" reader so you can have the extents of the envelope set via attributes created beforehand.

nielsgerrits
Rank
Userlevel 6
Badge +32

Use a search envelope in the reader parameters:

search envelope LASYou can also chose to use a FeatureReader instead of a "classic" reader so you can have the extents of the envelope set via attributes created beforehand.

The search envelope in a classic reader is the way to go indeed.

 

This functionality (the search envelope) is not available in the FeatureReader as far as I know.

The Spatial Filter in the FeatureReader also does not work for LAS in the FeatureReader.

 

This is one of the very few differences between classic readers and FeatureReaders, so I submitted a case for this.

M
Badge +2

@james_c_452​ You should be able to use the FeatureReader if the Initiator feature is a bounding box. You can then set the Spatial Filter parameter

james_c_452
Rank
Badge +4

Sorry, I should have been more specific. I am aware how to globally clip as everyone suggested (thanks) what I'd like to do is use a more piecemeal approach. Say that I have 10 tiles in a line and for a particular task I only need data for 3 randomly distributed specific spots. I haven't been able to work out how to do it in a way that only loads the relevant parts of the group of tiles. Though I haven't tried making a custom transformer for the feature reader, making it parallel by the area of interest, and then using the attribute values for the the bounding box of the area of interest.

 

Also, has anyone actually tried clipping a point cloud using the feature reader. If you have, does it first read the point cloud and then clip it, or does it only read what's in the bounding box (more what I'm after to save reading a lot of unnecessary data)?

caracadrian
Rank
Badge +20

Sorry, I should have been more specific. I am aware how to globally clip as everyone suggested (thanks) what I'd like to do is use a more piecemeal approach. Say that I have 10 tiles in a line and for a particular task I only need data for 3 randomly distributed specific spots. I haven't been able to work out how to do it in a way that only loads the relevant parts of the group of tiles. Though I haven't tried making a custom transformer for the feature reader, making it parallel by the area of interest, and then using the attribute values for the the bounding box of the area of interest.

 

Also, has anyone actually tried clipping a point cloud using the feature reader. If you have, does it first read the point cloud and then clip it, or does it only read what's in the bounding box (more what I'm after to save reading a lot of unnecessary data)?

From what I can tell, FeatureReader first reads the pointcloud and then applies the envelope (actually, filters WHILE reading but has to read the whole file).

If you need filtering before reading I recommend using database formats for storing data (any type of data). These are the only formats that can filter before reading/loading. I recommend this article as a starting point: Tutorial: Let the Database Do the Work (safe.com)

nielsgerrits
Rank
Userlevel 6
Badge +32

From what I can tell, FeatureReader first reads the pointcloud and then applies the envelope (actually, filters WHILE reading but has to read the whole file).

If you need filtering before reading I recommend using database formats for storing data (any type of data). These are the only formats that can filter before reading/loading. I recommend this article as a starting point: Tutorial: Let the Database Do the Work (safe.com)

From what I see the LAS file is read completely when using a FeatureReader with a Spatial Filter, not being clipped at all. Using FME 2021.2

nielsgerrits
Rank
Userlevel 6
Badge +32

@james_c_452​ You should be able to use the FeatureReader if the Initiator feature is a bounding box. You can then set the Spatial Filter parameter

Unfortunately this seems not to work, checked in 2021.2.

caracadrian
Rank
Badge +20

From what I can tell, FeatureReader first reads the pointcloud and then applies the envelope (actually, filters WHILE reading but has to read the whole file).

If you need filtering before reading I recommend using database formats for storing data (any type of data). These are the only formats that can filter before reading/loading. I recommend this article as a starting point: Tutorial: Let the Database Do the Work (safe.com)

In build 21784, setting Search Envelope via _xmin, _xmax, _ymin, _ymax attributes from BoundsExtractor clips the .las file in FeatureReader.

nielsgerrits
Rank
Userlevel 6
Badge +32

From what I can tell, FeatureReader first reads the pointcloud and then applies the envelope (actually, filters WHILE reading but has to read the whole file).

If you need filtering before reading I recommend using database formats for storing data (any type of data). These are the only formats that can filter before reading/loading. I recommend this article as a starting point: Tutorial: Let the Database Do the Work (safe.com)

Can you please show me how? With a print screen?

I can't find a Search Envelope option in the FeatureReader Parameters when using LAS and would like to learn how to reproduce this.

mark2atsafe
Rank
Userlevel 4
Badge +25

Knowing what I do about FME, I wonder if the data is even read until some sort of action occurs. We'd probably just read the header, and that includes the min and max x/y.

 

For example, if you read one of your 10gb point clouds into FME - say just a reader connected to a Junction - and you have caching turned off, then how long does it take? Admittedly, my point clouds are tiny compared to yours (only 8m points), but for me it only takes 0.3 seconds.

 

If I clip then that with a Clipper entirely unrelated then it still takes 0.3 seconds. Only when the actual Clipper overlaps does it have to "read" the data to clip it (2.9 seconds).

 

What I'm saying is, if you can turn off caching, then it may be quicker to just read all of the point clouds and cut them with a Clipper. Or, the Search Envelope tool is using the same trick, so that it too performs quickly and doesn't actually read data it knows won't overlap.

 

danminneyatsaf
Rank
Userlevel 2
Badge +10

From what I can tell, FeatureReader first reads the pointcloud and then applies the envelope (actually, filters WHILE reading but has to read the whole file).

If you need filtering before reading I recommend using database formats for storing data (any type of data). These are the only formats that can filter before reading/loading. I recommend this article as a starting point: Tutorial: Let the Database Do the Work (safe.com)

@nielsgerrits​ 

The real reason the result from the Spatial Filter looks wrong is because the spatial filter is inside of what is an entire pointcloud feature. Thus the default spatial intersection does an intersect on the whole feature (the entire point cloud) and the whole feature is returned.

 

The difference between this FeatureReader case and that in the regular reader is whether the Clip to Envelope is enabled (which actually does not exist on the FeatureReader). At the moment we are looking to implement this functionality for the FeatureReader.

mark2atsafe
Rank
Userlevel 4
Badge +25

Knowing what I do about FME, I wonder if the data is even read until some sort of action occurs. We'd probably just read the header, and that includes the min and max x/y.

 

For example, if you read one of your 10gb point clouds into FME - say just a reader connected to a Junction - and you have caching turned off, then how long does it take? Admittedly, my point clouds are tiny compared to yours (only 8m points), but for me it only takes 0.3 seconds.

 

If I clip then that with a Clipper entirely unrelated then it still takes 0.3 seconds. Only when the actual Clipper overlaps does it have to "read" the data to clip it (2.9 seconds).

 

What I'm saying is, if you can turn off caching, then it may be quicker to just read all of the point clouds and cut them with a Clipper. Or, the Search Envelope tool is using the same trick, so that it too performs quickly and doesn't actually read data it knows won't overlap.

 

In case it helps, I picked this as one of our questions of the week and covered it here: https://www.youtube.com/watch?v=ZvSaHQ7gouI&t=750s

nielsgerrits
Rank
Userlevel 6
Badge +32

From what I can tell, FeatureReader first reads the pointcloud and then applies the envelope (actually, filters WHILE reading but has to read the whole file).

If you need filtering before reading I recommend using database formats for storing data (any type of data). These are the only formats that can filter before reading/loading. I recommend this article as a starting point: Tutorial: Let the Database Do the Work (safe.com)

Ah thank you, I expected it to clip it as well indeed. I work a lot with vector data and expected it to filter pointcloud like it does points / lines / polygons. I just wanted to make sure I did not miss something.

caracadrian
Rank
Badge +20

From what I can tell, FeatureReader first reads the pointcloud and then applies the envelope (actually, filters WHILE reading but has to read the whole file).

If you need filtering before reading I recommend using database formats for storing data (any type of data). These are the only formats that can filter before reading/loading. I recommend this article as a starting point: Tutorial: Let the Database Do the Work (safe.com)

@nielsgerrits​ I mistook the FeatureReader for LAS Reader. The normal reader has a Search envelope and an option to clip while FeatureReader has a spatial filter but the point cloud is a single feature so it's not clipped.

Sorry about the misunderstanding.

I believe Safe should fix the inconsistencies between the different readers.

nielsgerrits
Rank
Userlevel 6
Badge +32

From what I can tell, FeatureReader first reads the pointcloud and then applies the envelope (actually, filters WHILE reading but has to read the whole file).

If you need filtering before reading I recommend using database formats for storing data (any type of data). These are the only formats that can filter before reading/loading. I recommend this article as a starting point: Tutorial: Let the Database Do the Work (safe.com)

No problem, I just wanted to make sure I understood correctly or missed something.

james_c_452
Rank
Badge +4

In case it helps, I picked this as one of our questions of the week and covered it here: https://www.youtube.com/watch?v=ZvSaHQ7gouI&t=750s

Thanks for the reply Mark, sorry I took so long to get back to you. It seems, like you're video, that the large clouds I use load reasonably quick. I think the problem might be that I mostly do operations straight away in my workspaces so the entire cloud needs to be loaded. In future I'll try using a clipper straight away to make it more efficient, at least when I can. I guess the problem is a lot of the time I need to do something on most of my data and preserve it as a point cloud!

Reply


Community Stats

28,684
Posts
109,411
Replies
36,542
Members
Terms & ConditionsCookie settings
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.