I have a file with a huge number of feature (points) and I want to affect random values for them. I use the TrueRandomIntGenerator transformer to do that, but this transformer is limited (no more than 10000 feature). How can I do to handle the file by blocks of 10000 elements?
Solved
Generating large amounts of random numbers
- Contributor
Hello,
Best answer by geosander
It could actually handle more than 10,000 features (as there is no limit built-in to the transformer, it seems), but since the transformer uses a service called Random, that will cause the actual limitation. These limitations are mentioned in the link in the transformer's documentation.
View originalThe number of random integers you can generate depends entirely on the settings of the transformer. I just tried it out with 10,000 features, using base 10, 0 as minimum and 4,000,000 as maximum and I have now used up almost half my bit allowance (1,000,000 originally), so I could even run it again probably.
The problem here is you cannot fool that service (since it registers incoming requests by IP address), so you either need to:
- Wait until your bit allowance has been topped up (+200,000 daily);
- Divide your requests over different machines (each with their own IP - might need to have different (sub)networks too), although that probably is illegal if they find out;
- Buy credits on Random.org so you can make all the requests you need.
<strong>This post is closed to further activity.</strong><br />
It may be a question with a best answer, an implemented idea, or just a post needing no comment.<br />
If you have a follow-up or related question, please <a href="https://community.safe.com/topic/new">post a new question or idea</a>.<br />
If there is a genuine update to be made, please contact us and request that the post is reopened.
+7It could actually handle more than 10,000 features (as there is no limit built-in to the transformer, it seems), but since the transformer uses a service called Random, that will cause the actual limitation. These limitations are mentioned in the link in the transformer's documentation.
The number of random integers you can generate depends entirely on the settings of the transformer. I just tried it out with 10,000 features, using base 10, 0 as minimum and 4,000,000 as maximum and I have now used up almost half my bit allowance (1,000,000 originally), so I could even run it again probably.
The problem here is you cannot fool that service (since it registers incoming requests by IP address), so you either need to:
- Wait until your bit allowance has been topped up (+200,000 daily);
- Divide your requests over different machines (each with their own IP - might need to have different (sub)networks too), although that probably is illegal if they find out;
- Buy credits on Random.org so you can make all the requests you need.
+44- Safer
geosander wrote:
It could actually handle more than 10,000 features (as there is no limit built-in to the transformer, it seems), but since the transformer uses a service called Random, that will cause the actual limitation. These limitations are mentioned in the link in the transformer's documentation.
The number of random integers you can generate depends entirely on the settings of the transformer. I just tried it out with 10,000 features, using base 10, 0 as minimum and 4,000,000 as maximum and I have now used up almost half my bit allowance (1,000,000 originally), so I could even run it again probably.
The problem here is you cannot fool that service (since it registers incoming requests by IP address), so you either need to:
- Wait until your bit allowance has been topped up (+200,000 daily);
- Divide your requests over different machines (each with their own IP - might need to have different (sub)networks too), although that probably is illegal if they find out;
- Buy credits on Random.org so you can make all the requests you need.
Very good answer. The only thing I'd add is that the user could use the plain RandomNumberGenerator transformer instead. It won't be truly as random as the web service, but it won't have the same limitations.
Hello,
As sander_s said, if you are at the point you have large enough quota at random.org to process millions of features, here's how to managed that using the TrueRandomIntGenerator transformer.
First, edit the TrueRandomIntGenerator transformer and create a user parameter for the "Parallel Process By" parameter.
Assuming that you have a valid quota for random.org, you'll be able to process as many features as you need.
Larry
+7larry wrote:
Hello,
As sander_s said, if you are at the point you have large enough quota at random.org to process millions of features, here's how to managed that using the TrueRandomIntGenerator transformer.
First, edit the TrueRandomIntGenerator transformer and create a user parameter for the "Parallel Process By" parameter.
Assuming that you have a valid quota for random.org, you'll be able to process as many features as you need.
Larry
Thanks for adding step 2 :)
Hi,
If the requirement allows using pseudo-random numbers, Python scripting could also be a workaround.
Takashi
- Contributor
Thank you for all your answers. I appreciate your reactivity and I am going to try hard to put into practice your advices.
takashi wrote:
Hi,
If the requirement allows using pseudo-random numbers, Python scripting could also be a workaround.
Takashi
Agreed, and it will also be a LOT faster, compared to fetching series from random.org.
Helpful Members This Week
Enter your E-mail address. We'll send you an e-mail with instructions to reset your password.