zodb-cache-size-bytes setting in plone.recipe.zope2instance

[Timo Stollenwerk]

Hi,

I was wondering about the status of the "zodb-cache-size-bytes" setting
in plone.recipe.zope2instance. Restricting zodb memory usage by memory
size instead of restricting it by the number of objects in the ZODB
seems like a sensible thing to do.

The pypi page says that this feature is still experimental:

plone.recipe.zope2instance: Set the ZODB cache sizes in bytes. This
feature is still experimental.

http://pypi.python.org/pypi/plone.recipe.zope2instance/4.2.1       


It seems like this feature has been merged into ZODB 3.9:

https://dev.plone.org/ticket/10238

https://github.com/plone/plone.recipe.zope2instance/blob/master/CHANGES.txt

- Add support for zodb-cache-size-bytes from ZODB 3.9 and later.
  [wichert]


The setting is used and properly written into the zope.conf:

https://github.com/plone/plone.recipe.zope2instance/blob/master/src/plone/recipe/zope2instance/__init__.py#L378


I would like to know if people are using it in production and if it
works as expected. If this is true, we might want to remove the
experimental hint from the pypi page and make this the recommended way.

Cheers,
timo

------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and
threat landscape has changed and how IT managers can respond. Discussions
will include endpoint security, mobile security and the latest in malware
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
_______________________________________________
Plone-developers mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/plone-developers

[Alan Runyan-3]

> I would like to know if people are using it in production and if it
> works as expected. If this is true, we might want to remove the
> experimental hint from the pypi page and make this the recommended way.

Yes.  We had some discussions internally about this last week.  Few notes:

1. It is only an estimation it is not possible for it to be accurate

2. It could very well be WILDLY inaccurate for 64bit platforms. would be
    great if someone could verify accuracy between 32/64bit platforms.

--
Alan Runyan

Skype/Twitter:: runyaga
Office:: 713.942.2377 ext 111
http://ploud.com/  Plone site in less than 10 seconds

------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and
threat landscape has changed and how IT managers can respond. Discussions
will include endpoint security, mobile security and the latest in malware
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
_______________________________________________
Plone-developers mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/plone-developers

[hvelarde]

don't you think it would be good also to set default value of
zodb-cache-size to a number larger than 10,000 objects?

memory is cheap these days and sites are larger.

it's amazing how many Plone sites are out there having performance
issues because this number is too low.

best regards

Héctor Velarde


------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and
threat landscape has changed and how IT managers can respond. Discussions
will include endpoint security, mobile security and the latest in malware
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
_______________________________________________
Plone-developers mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/plone-developers

[Hanno Schlichting-4]

Héctor Velarde <hector.velarde@...> writes:
> don't you think it would be good also to set default value of
> zodb-cache-size to a number larger than 10,000 objects?
>
> memory is cheap these days and sites are larger.
>
> it's amazing how many Plone sites are out there having performance
> issues because this number is too low.

I think that's worth considering. I increased the value from a default 5000 to
10000 back in 2009 for the first Plone 4.0 alpha release. At that point I took
neither blobs nor Python 2.6's ability to give memory back to the OS into
account.

Now you get most large content in blobs, which keeps it outside the ZODB cache.
And memory use shrinks again after some peak load. My rule of thumb for Plone 3
was to calculate with 50-100kb per object, for Plone 4 it's 4-8kb per object.
And then there's a base memory use of about 100mb for Plone itself. All on
32bit systems, 64bit is ~1.5 the size. Those numbers could use some larger
sample size to validate them ;-)

For a "small" site (up to a couple hundred content items), changing these
values should have no impact, as they don't ever reach the current cache size.

Let's assume some "medium" sized site, where there's enough content to fill the
caches no matter what size they are. I think you need in the order of thousands
of content items to get into this category. My definition of large site begins
with one hundred thousand content items, at which point you have multiple
servers and need to tune various bits of the operating system and stack, so
the default value doesn't apply.

If my numbers are correct, we currently get:

Minimum (32bit): 100mb base + 10000 items * 2 threads * 4kb ~= 180mb
Maximum (64bit): 150mb base + 10000 items * 2 threads * 12kb ~= 380mb

Let's say we triple the size to 30000 items:

Minimum (32bit): 100mb base + 30000 items * 2 threads * 4kb ~= 330mb
Maximum (64bit): 150mb base + 30000 items * 2 threads * 12kb ~= 850mb

Or go all the way up to 50000 items:

Minimum (32bit): 100mb base + 50000 items * 2 threads * 4kb ~= 490mb
Maximum (64bit): 150mb base + 50000 items * 2 threads * 12kb ~= 1.3gb

Those numbers are per process, and there's likely at least two processes
per machine for "medium" sites.

Looking at the typical Rackspace and Amazon EC2 machine types, I'd expect at
least 2gb or 4gb machines to be used for "medium" Plone sites these days.

Taking all those numbers into account, I think increasing the default cache
size to 30000 should be safe. At 50000 the chance to hit the typical memory
limit and start swapping gets much bigger. And if someone isn't adjusting the
ZODB cache size, there's likely nobody around to monitor memory usage and
swapping either.

Other opinions and "statistics" welcome ;)

Hanno


------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and
threat landscape has changed and how IT managers can respond. Discussions
will include endpoint security, mobile security and the latest in malware
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
_______________________________________________
Plone-developers mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/plone-developers

[Hanno Schlichting-4]

Timo Stollenwerk <lists@...> writes:
> I was wondering about the status of the "zodb-cache-size-bytes" setting
> in plone.recipe.zope2instance. Restricting zodb memory usage by memory
> size instead of restricting it by the number of objects in the ZODB
> seems like a sensible thing to do.

Well yes, but what's actually tracked is the byte size of the pickle data and
not memory size. Pickle size and memory size are correlated, but the ratio
depends on each Python class and a number of system properties: namely 32/64bit
for numbers and UCS-2/4 Unicode builds for strings.

As an extreme example, take the former DateTime 2. It has a pickle size of
about 400 bytes. On a 32bit/UCS-2 system it uses 2kb of memory. On a
64bit/UCS-4 it uses 4kb. So you get a worst case factor of 10 times. The newer
DateTime 3 has a factor of two (32bit) or three times (64bit). Probably most
common Python classes will have a factor of 2-4. Archetypes BaseUnit's storing
the real text data should be dominated by the size of the text, so for any
ASCII letter you get either 2 or 4 bytes depending on the Unicode build.

With all that said, you could advertise this setting and note that you should
set it to 1/4 of the target memory size, not forgetting to mention threads.

I think someone needs to try this on a variety of different sites and see if
such a rule is anywhere close at all. Otherwise you just replace one
intractable but well proven setting with another equally obscure one.

> The pypi page says that this feature is still experimental:
>
> I would like to know if people are using it in production and if it
> works as expected. If this is true, we might want to remove the
> experimental hint from the pypi page and make this the recommended way.

I tried to use it back in 2010, but hit too many bugs. I don't remember if me
and Nikolay from Enfold actually fixed all of them in the end. I never tried it
in production and to my knowledge Enfold is the only company that did.

So I wouldn't remove the experimental flag unless there's a lot more people
reporting success with this.

Hanno


------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and
threat landscape has changed and how IT managers can respond. Discussions
will include endpoint security, mobile security and the latest in malware
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
_______________________________________________
Plone-developers mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/plone-developers

[hvelarde]

On 28/04/12 22:43, Hanno Schlichting wrote:
> Taking all those numbers into account, I think increasing the default cache
> size to 30000 should be safe. At 50000 the chance to hit the typical memory
> limit and start swapping gets much bigger. And if someone isn't adjusting the
> ZODB cache size, there's likely nobody around to monitor memory usage and
> swapping either.

thanks for the info, Hanno!

funny that this was the number I had in mind without making any
calculations :-p

how can we make it happen? sometimes I see things being discussed but no
action taken.

best regards


------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and
threat landscape has changed and how IT managers can respond. Discussions
will include endpoint security, mobile security and the latest in malware
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
_______________________________________________
Plone-developers mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/plone-developers

[David Glick (GW)]

On May 1, 2012, at 11:01 AM, Héctor Velarde wrote:

> On 28/04/12 22:43, Hanno Schlichting wrote:
>> Taking all those numbers into account, I think increasing the default cache
>> size to 30000 should be safe. At 50000 the chance to hit the typical memory
>> limit and start swapping gets much bigger. And if someone isn't adjusting the
>> ZODB cache size, there's likely nobody around to monitor memory usage and
>> swapping either.
>
> thanks for the info, Hanno!
>
> funny that this was the number I had in mind without making any
> calculations :-p
>
> how can we make it happen? sometimes I see things being discussed but no
> action taken.
>

Check out the code, make the change, check it in…is usually the best way :)
David



----------
David Glick
 Web Developer
 [hidden email]
 206.286.1235x32

GiveBIG is coming! Mark your calendar for May 2 and get ready to give big to Groundwire
on this community-wide day of giving.

http://www.seattlefoundation.org/npos/Pages/Groundwire.aspx



------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and
threat landscape has changed and how IT managers can respond. Discussions
will include endpoint security, mobile security and the latest in malware
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
_______________________________________________
Plone-developers mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/plone-developers

[hvelarde]

On 01/05/12 15:09, David Glick (GW) wrote:
> Check out the code, make the change, check it in…is usually the best way :)

done! :-)

https://github.com/plone/plone.recipe.zope2instance/commit/03528d772ca02ca9011c15062ed75f942ec2bb9f


------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and
threat landscape has changed and how IT managers can respond. Discussions
will include endpoint security, mobile security and the latest in malware
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
_______________________________________________
Plone-developers mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/plone-developers