Don't use range validator...

[Volnei]

... especially for large ranges!

Hi all,

I'm writing just to share with you my terrible experience with the range
validator.
I and my team are working on a large project myself with hundreds of domain
classes, all with their validators and we are very careful in assigning
them.

Well, part of this project was to produce in the last week and to our
surprise it was a terrible experience. Especially when having a transaction
insert into one of our domain classes . The server CPU reached 100 % and
stayed there while the performance falls every minute. I confess that I
felt frustrated.
We started investigating the causes of slowdowns and after exhausting any
possibility of the database, triggers, constraints, retentions, locks and
more... imagine that it could be something related to the behavior of
hibernate.
Again we did not have any developments, we read several times the GORM
Gotchas and some very good performances from Burt Beckwith, but we were
still with the problem .
The light at the end of the tunnel appeared when disabling validation with
a save (validate: false) and the application started to respond with a
satisfactory time. But did not make sense that some constraints could
generate so much delay.
Then investigated constraint by constraint and disabling one by one to
identify what was being more costly and came to the conclusion that it was
a range constraint we had in a field of a domain class. Something like this:

field range : 0 .. 9999999999

No biggie right? Wrong!

We understand what the operation of the range and to our surprise is not
something simple like a:

it> from && it < to

The Range of the Groovy class actually creates a list and runs the same to
see if the value is valid . Understand that I am not questioning the
operation of the same , only saying that her performance can be very
degrading . The following script takes only 1 milesegundo to run:

def r = 1 .. 9999999999
println r.contains ( 1 )

The same script to a larger value takes more than 1 second.

def r = 1..9999999999
println r.contains(20000000)

And with the maximum value takes up to 5 minutes.

Not terrible? After removing all constraints of range back to being
satisfied with this framework that fills my eyes. And that does not pass
through the same situation, I decided to share with you.

Enjoy!

[Aaron Long]

I think this is a problem with your code. The value "9999999999" is >
Integer.MAX_VALUE and therefore causing the Groovy compiler to have to do
some weird stuff when used as the end of the range.

For instance:

def r = 1..Integer.MAX_VALUE
println r.contains(20000000)

returns "true" in milliseconds as does:

def r = 1..2147483647 // MAX Integer value
println r.contains(20000000)

However,

def r = 1..2147483648 // MAX + 1
println r.contains(20000000)

returns "false" because it ends up being 0 (or maybe -1, my two's
complement is bad).

Anyway, if you use 1..Integer.MAX_VALUE as your upper range, you shouldn't
have performance issues.

-Aaron

Post by Volnei
... especially for large ranges!
Hi all,
I'm writing just to share with you my terrible experience with the range
validator.
I and my team are working on a large project myself with hundreds of
domain classes, all with their validators and we are very careful in
assigning them.
Well, part of this project was to produce in the last week and to our
surprise it was a terrible experience. Especially when having a transaction
insert into one of our domain classes . The server CPU reached 100 % and
stayed there while the performance falls every minute. I confess that I
felt frustrated.
We started investigating the causes of slowdowns and after exhausting any
possibility of the database, triggers, constraints, retentions, locks and
more... imagine that it could be something related to the behavior of
hibernate.
Again we did not have any developments, we read several times the GORM
Gotchas and some very good performances from Burt Beckwith, but we were
still with the problem .
The light at the end of the tunnel appeared when disabling validation with
a save (validate: false) and the application started to respond with a
satisfactory time. But did not make sense that some constraints could
generate so much delay.
Then investigated constraint by constraint and disabling one by one to
identify what was being more costly and came to the conclusion that it was
field range : 0 .. 9999999999
No biggie right? Wrong!
We understand what the operation of the range and to our surprise is not
it> from && it < to
The Range of the Groovy class actually creates a list and runs the same to
see if the value is valid . Understand that I am not questioning the
operation of the same , only saying that her performance can be very
def r = 1 .. 9999999999
println r.contains ( 1 )
The same script to a larger value takes more than 1 second.
def r = 1..9999999999
println r.contains(20000000)
And with the maximum value takes up to 5 minutes.
Not terrible? After removing all constraints of range back to being
satisfied with this framework that fills my eyes. And that does not pass
through the same situation, I decided to share with you.
Enjoy!

[Volnei]

So if the follow code returns "false"

def r = 1..2147483648 // MAX + 1
println r.contains(20000000)

Should follow code also returns "false", right?

def r = 1..99999999999999
println r.contains(2)

But it returns "true"




To me, there something wrong in this code:

def r = 1..9999999999
println r.contains(2000000)

def r1 = 1..Integer.MAX_VALUE
println r1.contains(2000000)

def r2 = 1..Integer.MAX_VALUE + 1
println r2.contains(2000000)

def r3 = 1..2147483647 + 1
println r3.contains(2000000)

def r4 = 1..2147483648
println r4.contains(2000000)


So as I told the "large ranges" can cause problems.

I'm just alerting people who can have the same problem with no compiler or
runtime exceptions.

Thanks


________________________________________
Volnei Granado Munhoz

Post by Aaron Long
I think this is a problem with your code. The value "9999999999" is >
Integer.MAX_VALUE and therefore causing the Groovy compiler to have to do
some weird stuff when used as the end of the range.
def r = 1..Integer.MAX_VALUE
println r.contains(20000000)
def r = 1..2147483647 // MAX Integer value
println r.contains(20000000)
However,
def r = 1..2147483648 // MAX + 1
println r.contains(20000000)
returns "false" because it ends up being 0 (or maybe -1, my two's
complement is bad).
Anyway, if you use 1..Integer.MAX_VALUE as your upper range, you shouldn't
have performance issues.
-Aaron

Post by Volnei
... especially for large ranges!
Hi all,
I'm writing just to share with you my terrible experience with the range
validator.
I and my team are working on a large project myself with hundreds of
domain classes, all with their validators and we are very careful in
assigning them.
Well, part of this project was to produce in the last week and to our
surprise it was a terrible experience. Especially when having a transaction
insert into one of our domain classes . The server CPU reached 100 % and
stayed there while the performance falls every minute. I confess that I
felt frustrated.
We started investigating the causes of slowdowns and after exhausting any
possibility of the database, triggers, constraints, retentions, locks and
more... imagine that it could be something related to the behavior of
hibernate.
Again we did not have any developments, we read several times the GORM
Gotchas and some very good performances from Burt Beckwith, but we were
still with the problem .
The light at the end of the tunnel appeared when disabling validation
with a save (validate: false) and the application started to respond with a
satisfactory time. But did not make sense that some constraints could
generate so much delay.
Then investigated constraint by constraint and disabling one by one to
identify what was being more costly and came to the conclusion that it was
field range : 0 .. 9999999999
No biggie right? Wrong!
We understand what the operation of the range and to our surprise is not
it> from && it < to
The Range of the Groovy class actually creates a list and runs the same
to see if the value is valid . Understand that I am not questioning the
operation of the same , only saying that her performance can be very
def r = 1 .. 9999999999
println r.contains ( 1 )
The same script to a larger value takes more than 1 second.
def r = 1..9999999999
println r.contains(20000000)
And with the maximum value takes up to 5 minutes.
Not terrible? After removing all constraints of range back to being
satisfied with this framework that fills my eyes. And that does not pass
through the same situation, I decided to share with you.
Enjoy!

[Aaron Long]

This is generally true of anything if you use a value larger the
Integer.MAX_VALUE for something that holds an Integer (Range in Groovy is
actually an IntRange implementation, I believe).

Type this into your groovyConsole:

int foo = 99999999999999

and you'll see that the result is actually 276447231. This is essentially
what you get when you try to stuff a huge number into a signed 32-bit field.

I understand what you are saying, I just wanted to point out that it's
really a misuse of the range that is causing the performance problems.

-Aaron

[Jeff Scott Brown]

Post by Aaron Long
This is generally true of anything if you use a value larger the
Integer.MAX_VALUE for something that holds an Integer (Range in Groovy is
actually an IntRange implementation, I believe).
int foo = 99999999999999
and you'll see that the result is actually 276447231. This is essentially
what you get when you try to stuff a huge number into a signed 32-bit field.
I understand what you are saying, I just wanted to point out that it's
really a misuse of the range that is causing the performance problems.
-Aaron

Can you guys take this to the new Google group or the Groovy mailing list?  We have deprecated the Grails mailing lists.  See http://grails.1312388.n4.nabble.com/Deprecating-The-Grails-Mailing-Lists-td4656735.html.

Thanks.



JSB
—
Jeff Scott Brown
***@gopivotal.com

Find The Cause ~ Find The Cure
http://www.autismspeaks.org/



---------------------------------------------------------------------
To unsubscribe from this list, please visit:

http://xircles.codehaus.org/manage_email

[Volnei]

George, yes! I actually do this, the porpouse for this post is just to
alert others.

Jeff, sute!!

Thanks..

________________________________________
Volnei Granado Munhoz

Post by Aaron Long

Post by Aaron Long
This is generally true of anything if you use a value larger the
Integer.MAX_VALUE for something that holds an Integer (Range in Groovy is
actually an IntRange implementation, I believe).
int foo = 99999999999999
and you'll see that the result is actually 276447231. This is essentially
what you get when you try to stuff a huge number into a signed 32-bit

field.

Post by Aaron Long
I understand what you are saying, I just wanted to point out that it's
really a misuse of the range that is causing the performance problems.
-Aaron

Can you guys take this to the new Google group or the Groovy mailing list?
We have deprecated the Grails mailing lists. See
http://grails.1312388.n4.nabble.com/Deprecating-The-Grails-Mailing-Lists-td4656735.html
.
Thanks.
JSB
—
Jeff Scott Brown
Find The Cause ~ Find The Cure
http://www.autismspeaks.org/
---------------------------------------------------------------------
http://xircles.codehaus.org/manage_email

[Volnei]

Aaron,

I agree with you, is a misuse! Yes it's a IntRange but it also handle Long
values... try this

def range = 1..9999999999999
println range.to.class

Now the to is a Long with 9999999999999 value.

See you in the Google Group.

Thanks!



________________________________________
Volnei Granado Munhoz

Post by Volnei
George, yes! I actually do this, the porpouse for this post is just to
alert others.
Jeff, sute!!
Thanks..
________________________________________
Volnei Granado Munhoz

Post by Aaron Long

Post by Aaron Long
This is generally true of anything if you use a value larger the
Integer.MAX_VALUE for something that holds an Integer (Range in Groovy

is

Post by Aaron Long
actually an IntRange implementation, I believe).
int foo = 99999999999999
and you'll see that the result is actually 276447231. This is

essentially

Post by Aaron Long
what you get when you try to stuff a huge number into a signed 32-bit

field.

Post by Aaron Long
I understand what you are saying, I just wanted to point out that it's
really a misuse of the range that is causing the performance problems.
-Aaron

Can you guys take this to the new Google group or the Groovy mailing
list? We have deprecated the Grails mailing lists. See
http://grails.1312388.n4.nabble.com/Deprecating-The-Grails-Mailing-Lists-td4656735.html
.
Thanks.
JSB
—
Jeff Scott Brown
Find The Cause ~ Find The Cure
http://www.autismspeaks.org/
---------------------------------------------------------------------
http://xircles.codehaus.org/manage_email