Plone › General Questions

Performance problems

Classic

List

Threaded

23 messages Options

Fabian Büchler

Performance problems

Hello everyone,

our company develops / maintains the official tourism website for Vienna.

We are experiencing performance problems for quite some time now, but
increasing over the last weeks, and hope to find some help here.

We believe in some bottleneck at ZEO or ZODB level, but are no experts
in tuning both.
That's because the problems usually occur whenever lots of editors are
working at the same time and we are having an increased editing load
in the last weeks.

Can anyone provide us with some advice about zodb/zeo tuning
parameters or any good tools to get some useful metrics?
Also, if anyone knows (or is) experts who do (paid) consulting on that
topic, please feel free to refer them to me.

Thanks for any help in advance!
Regards, Fabian

Some insight into our system:

* It's a very complex Plone setup with lots of custom modifications to
Plone (used as a editing backend) and a custom frontend.
We are serving content for a couple of different subsites, an API and more.

* Currently we are still running Plone 3.2. (an upgrade to Plone 4.1
is in preparation, but won't be done before a few months from now)

* The installation is running on two dedicated servers (which are also
serving one smaller django installation and some static stuff), both
16 GB ram, 2x 2 GHz dualcore processors, powered by CentOS,...

* per month we have about 300.000 to 500.000 unique users, 3 to 12
mio. pageviews (GA vs. awstats), 1200 GB traffic,...

* the system setup is
* both servers run lighttpd > varnish > 3 zope instances / zeo clients
* one server runs zeo, and the zodb (currently about 2 GB, 10 GB
blobstorage) is on a DRBD shared storage

* server metrics say, that the servers are not overloaded. CPU usage
between 30% and 80% at peak times, I/O far from any critical numbers.

anything else you need to know?

------------------------------------------------------------------------------
Learn Windows Azure Live! Tuesday, Dec 13, 2011
Microsoft is holding a special Learn Windows Azure training event for
developers. It will provide a great way to learn Windows Azure and what it
provides. You can attend the event by watching it streamed LIVE online.
Learn more at http://p.sf.net/sfu/ms-windowsazure
_______________________________________________
Plone-Users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/plone-users

Remove Ads

Nathan Van Gheem

Re: Performance problems

Have you tweaked the zeo client cache settings zodb-cache-size option in http://pypi.python.org/pypi/plone.recipe.zope2instance/4.2 recipe?

How large is your portal_catalog? Have you added a good deal of indexes and metadata? Mounting the catalog to it's own mount point helps out also.

Take a look at http://scalingplone.pbworks.com/w/page/3679597/FrontPage for further tips.

On Fri, Dec 16, 2011 at 8:22 AM, Fabian Büchler <[hidden email]> wrote:

Hello everyone,

our company develops / maintains the official tourism website for Vienna.

We are experiencing performance problems for quite some time now, but
increasing over the last weeks, and hope to find some help here.

We believe in some bottleneck at ZEO or ZODB level, but are no experts
in tuning both.
That's because the problems usually occur whenever lots of editors are
working at the same time and we are having an increased editing load
in the last weeks.

Can anyone provide us with some advice about zodb/zeo tuning
parameters or any good tools to get some useful metrics?
Also, if anyone knows (or is) experts who do (paid) consulting on that
topic, please feel free to refer them to me.

Thanks for any help in advance!
Regards, Fabian

Some insight into our system:

* It's a very complex Plone setup with lots of custom modifications to
Plone (used as a editing backend) and a custom frontend.
We are serving content for a couple of different subsites, an API and more.

* Currently we are still running Plone 3.2. (an upgrade to Plone 4.1
is in preparation, but won't be done before a few months from now)

* The installation is running on two dedicated servers (which are also
serving one smaller django installation and some static stuff), both
16 GB ram, 2x 2 GHz dualcore processors, powered by CentOS,...

* per month we have about 300.000 to 500.000 unique users, 3 to 12
mio. pageviews (GA vs. awstats), 1200 GB traffic,...

* the system setup is
* both servers run lighttpd > varnish > 3 zope instances / zeo clients
* one server runs zeo, and the zodb (currently about 2 GB, 10 GB
blobstorage) is on a DRBD shared storage

* server metrics say, that the servers are not overloaded. CPU usage
between 30% and 80% at peak times, I/O far from any critical numbers.

anything else you need to know?

------------------------------------------------------------------------------
Learn Windows Azure Live! Tuesday, Dec 13, 2011
Microsoft is holding a special Learn Windows Azure training event for
developers. It will provide a great way to learn Windows Azure and what it
provides. You can attend the event by watching it streamed LIVE online.
Learn more at http://p.sf.net/sfu/ms-windowsazure
_______________________________________________
Plone-Users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/plone-users

------------------------------------------------------------------------------
Learn Windows Azure Live! Tuesday, Dec 13, 2011
Microsoft is holding a special Learn Windows Azure training event for
developers. It will provide a great way to learn Windows Azure and what it
provides. You can attend the event by watching it streamed LIVE online.
Learn more at http://p.sf.net/sfu/ms-windowsazure
_______________________________________________
Plone-Users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/plone-users

Fabian Büchler

Re: Performance problems

We are planning to do some iterative changes to zeo client cache size
(from 100MB to 2/300 MB) and the zodb-cache-size (from 25000 to 100000
objects) and the zserver-threads (from not set to 8-10) in the next
weeks (always with some time to monitor the effects).
Do you have any advice on some good numbers there?

The portal_catalog says it holds 42,347 records. I don't know how many
objects the zodb holds in total. Do you know how to find out?
We've added only a few (~5) boolean values to the catalog, and 3-4
short strings, but nothing big.

With mounting the catalog to it's own moint point, you mean having a
second data.fs which holds just the catalog ?

Thanks for your quick answer!
Regards,
Fabian

2011/12/16 Nathan Van Gheem <[hidden email]>:

> Have you tweaked the zeo client cache settings zodb-cache-size option
> in http://pypi.python.org/pypi/plone.recipe.zope2instance/4.2 recipe?
>
> How large is your portal_catalog? Have you added a good deal of indexes and
> metadata? Mounting the catalog to it's own mount point helps out also.
>
> Take a look at http://scalingplone.pbworks.com/w/page/3679597/FrontPage for
> further tips.
>
> On Fri, Dec 16, 2011 at 8:22 AM, Fabian Büchler <[hidden email]>
> wrote:
>>
>> Hello everyone,
>>
>> our company develops / maintains the official tourism website for Vienna.
>>
>> We are experiencing performance problems for quite some time now, but
>> increasing over the last weeks, and hope to find some help here.
>>
>> We believe in some bottleneck at ZEO or ZODB level, but are no experts
>> in tuning both.
>> That's because the problems usually occur whenever lots of editors are
>> working at the same time and we are having an increased editing load
>> in the last weeks.
>>
>> Can anyone provide us with some advice about zodb/zeo tuning
>> parameters or any good tools to get some useful metrics?
>> Also, if anyone knows (or is) experts who do (paid) consulting on that
>> topic, please feel free to refer them to me.
>>
>> Thanks for any help in advance!
>> Regards, Fabian
>>
>>
>> Some insight into our system:
>>
>> * It's a very complex Plone setup with lots of custom modifications to
>> Plone (used as a editing backend) and a custom frontend.
>> We are serving content for a couple of different subsites, an API and
>> more.
>>
>> * Currently we are still running Plone 3.2. (an upgrade to Plone 4.1
>> is in preparation, but won't be done before a few months from now)
>>
>> * The installation is running on two dedicated servers (which are also
>> serving one smaller django installation and some static stuff), both
>> 16 GB ram, 2x 2 GHz dualcore processors, powered by CentOS,...
>>
>> * per month we have about 300.000 to 500.000 unique users, 3 to 12
>> mio. pageviews (GA vs. awstats), 1200 GB traffic,...
>>
>> * the system setup is
>> * both servers run lighttpd > varnish > 3 zope instances / zeo clients
>> * one server runs zeo, and the zodb (currently about 2 GB, 10 GB
>> blobstorage) is on a DRBD shared storage
>>
>> * server metrics say, that the servers are not overloaded. CPU usage
>> between 30% and 80% at peak times, I/O far from any critical numbers.
>>
>> anything else you need to know?
>>
>>
>> ------------------------------------------------------------------------------
>> Learn Windows Azure Live! Tuesday, Dec 13, 2011
>> Microsoft is holding a special Learn Windows Azure training event for
>> developers. It will provide a great way to learn Windows Azure and what it
>> provides. You can attend the event by watching it streamed LIVE online.
>> Learn more at http://p.sf.net/sfu/ms-windowsazure
>> _______________________________________________
>> Plone-Users mailing list
>> [hidden email]
>> https://lists.sourceforge.net/lists/listinfo/plone-users
>
>

Nathan Van Gheem

Re: Performance problems

zodb-cache-size (from 25000 to 100000

Cache as much as you can.

zserver-threads (from not set to 8-10)

This might be your problem. The more you have, the slower performance you get. Only use 1 or 2 and use more clients if you need better performance.

On Fri, Dec 16, 2011 at 9:33 AM, Fabian Büchler <[hidden email]> wrote:

We are planning to do some iterative changes to zeo client cache size
(from 100MB to 2/300 MB) and the zodb-cache-size (from 25000 to 100000
objects) and the zserver-threads (from not set to 8-10) in the next
weeks (always with some time to monitor the effects).
Do you have any advice on some good numbers there?

The portal_catalog says it holds 42,347 records. I don't know how many
objects the zodb holds in total. Do you know how to find out?
We've added only a few (~5) boolean values to the catalog, and 3-4
short strings, but nothing big.

With mounting the catalog to it's own moint point, you mean having a
second data.fs which holds just the catalog ?

Thanks for your quick answer!
Regards,
Fabian

2011/12/16 Nathan Van Gheem <[hidden email]>:

> Have you tweaked the zeo client cache settings zodb-cache-size option
> in http://pypi.python.org/pypi/plone.recipe.zope2instance/4.2 recipe?
>
> How large is your portal_catalog? Have you added a good deal of indexes and
> metadata? Mounting the catalog to it's own mount point helps out also.
>
> Take a look at http://scalingplone.pbworks.com/w/page/3679597/FrontPage for
> further tips.
>
> On Fri, Dec 16, 2011 at 8:22 AM, Fabian Büchler <[hidden email]>
> wrote:
>>
>> Hello everyone,
>>
>> our company develops / maintains the official tourism website for Vienna.
>>
>> We are experiencing performance problems for quite some time now, but
>> increasing over the last weeks, and hope to find some help here.
>>
>> We believe in some bottleneck at ZEO or ZODB level, but are no experts
>> in tuning both.
>> That's because the problems usually occur whenever lots of editors are
>> working at the same time and we are having an increased editing load
>> in the last weeks.
>>
>> Can anyone provide us with some advice about zodb/zeo tuning
>> parameters or any good tools to get some useful metrics?
>> Also, if anyone knows (or is) experts who do (paid) consulting on that
>> topic, please feel free to refer them to me.
>>
>> Thanks for any help in advance!
>> Regards, Fabian
>>
>>
>> Some insight into our system:
>>
>> * It's a very complex Plone setup with lots of custom modifications to
>> Plone (used as a editing backend) and a custom frontend.
>> We are serving content for a couple of different subsites, an API and
>> more.
>>
>> * Currently we are still running Plone 3.2. (an upgrade to Plone 4.1
>> is in preparation, but won't be done before a few months from now)
>>
>> * The installation is running on two dedicated servers (which are also
>> serving one smaller django installation and some static stuff), both
>> 16 GB ram, 2x 2 GHz dualcore processors, powered by CentOS,...
>>
>> * per month we have about 300.000 to 500.000 unique users, 3 to 12
>> mio. pageviews (GA vs. awstats), 1200 GB traffic,...
>>
>> * the system setup is
>> * both servers run lighttpd > varnish > 3 zope instances / zeo clients
>> * one server runs zeo, and the zodb (currently about 2 GB, 10 GB
>> blobstorage) is on a DRBD shared storage
>>
>> * server metrics say, that the servers are not overloaded. CPU usage
>> between 30% and 80% at peak times, I/O far from any critical numbers.
>>
>> anything else you need to know?
>>
>>
>> ------------------------------------------------------------------------------
>> Learn Windows Azure Live! Tuesday, Dec 13, 2011
>> Microsoft is holding a special Learn Windows Azure training event for
>> developers. It will provide a great way to learn Windows Azure and what it
>> provides. You can attend the event by watching it streamed LIVE online.
>> Learn more at http://p.sf.net/sfu/ms-windowsazure
>> _______________________________________________
>> Plone-Users mailing list
>> [hidden email]
>> https://lists.sourceforge.net/lists/listinfo/plone-users
>
>

Nathan Van Gheem

Re: Performance problems

With mounting the catalog to it's own moint point, you mean having a
second data.fs which holds just the catalog ?

Yes, you should be able to find info on how to do this on the interwebz :)

On Fri, Dec 16, 2011 at 9:46 AM, Nathan Van Gheem <[hidden email]> wrote:

zodb-cache-size (from 25000 to 100000

Cache as much as you can.

zserver-threads (from not set to 8-10)
This might be your problem. The more you have, the slower performance you get. Only use 1 or 2 and use more clients if you need better performance.

On Fri, Dec 16, 2011 at 9:33 AM, Fabian Büchler <[hidden email]> wrote:
We are planning to do some iterative changes to zeo client cache size
(from 100MB to 2/300 MB) and the zodb-cache-size (from 25000 to 100000
objects) and the zserver-threads (from not set to 8-10) in the next
weeks (always with some time to monitor the effects).
Do you have any advice on some good numbers there?

The portal_catalog says it holds 42,347 records. I don't know how many
objects the zodb holds in total. Do you know how to find out?
We've added only a few (~5) boolean values to the catalog, and 3-4
short strings, but nothing big.

With mounting the catalog to it's own moint point, you mean having a
second data.fs which holds just the catalog ?

Thanks for your quick answer!
Regards,
Fabian

2011/12/16 Nathan Van Gheem <[hidden email]>:

> Have you tweaked the zeo client cache settings zodb-cache-size option
> in http://pypi.python.org/pypi/plone.recipe.zope2instance/4.2 recipe?
>
> How large is your portal_catalog? Have you added a good deal of indexes and
> metadata? Mounting the catalog to it's own mount point helps out also.
>
> Take a look at http://scalingplone.pbworks.com/w/page/3679597/FrontPage for
> further tips.
>
> On Fri, Dec 16, 2011 at 8:22 AM, Fabian Büchler <[hidden email]>
> wrote:
>>
>> Hello everyone,
>>
>> our company develops / maintains the official tourism website for Vienna.
>>
>> We are experiencing performance problems for quite some time now, but
>> increasing over the last weeks, and hope to find some help here.
>>
>> We believe in some bottleneck at ZEO or ZODB level, but are no experts
>> in tuning both.
>> That's because the problems usually occur whenever lots of editors are
>> working at the same time and we are having an increased editing load
>> in the last weeks.
>>
>> Can anyone provide us with some advice about zodb/zeo tuning
>> parameters or any good tools to get some useful metrics?
>> Also, if anyone knows (or is) experts who do (paid) consulting on that
>> topic, please feel free to refer them to me.
>>
>> Thanks for any help in advance!
>> Regards, Fabian
>>
>>
>> Some insight into our system:
>>
>> * It's a very complex Plone setup with lots of custom modifications to
>> Plone (used as a editing backend) and a custom frontend.
>> We are serving content for a couple of different subsites, an API and
>> more.
>>
>> * Currently we are still running Plone 3.2. (an upgrade to Plone 4.1
>> is in preparation, but won't be done before a few months from now)
>>
>> * The installation is running on two dedicated servers (which are also
>> serving one smaller django installation and some static stuff), both
>> 16 GB ram, 2x 2 GHz dualcore processors, powered by CentOS,...
>>
>> * per month we have about 300.000 to 500.000 unique users, 3 to 12
>> mio. pageviews (GA vs. awstats), 1200 GB traffic,...
>>
>> * the system setup is
>> * both servers run lighttpd > varnish > 3 zope instances / zeo clients
>> * one server runs zeo, and the zodb (currently about 2 GB, 10 GB
>> blobstorage) is on a DRBD shared storage
>>
>> * server metrics say, that the servers are not overloaded. CPU usage
>> between 30% and 80% at peak times, I/O far from any critical numbers.
>>
>> anything else you need to know?
>>
>>
>> ------------------------------------------------------------------------------
>> Learn Windows Azure Live! Tuesday, Dec 13, 2011
>> Microsoft is holding a special Learn Windows Azure training event for
>> developers. It will provide a great way to learn Windows Azure and what it
>> provides. You can attend the event by watching it streamed LIVE online.
>> Learn more at http://p.sf.net/sfu/ms-windowsazure
>> _______________________________________________
>> Plone-Users mailing list
>> [hidden email]
>> https://lists.sourceforge.net/lists/listinfo/plone-users
>
>

David Glick (GW)

Re: Performance problems

In reply to this post by Fabian Büchler

On Dec 16, 2011, at 6:22 AM, Fabian Büchler wrote:

> Hello everyone,
>
> our company develops / maintains the official tourism website for Vienna.
>
> We are experiencing performance problems for quite some time now, but
> increasing over the last weeks, and hope to find some help here.
>
> We believe in some bottleneck at ZEO or ZODB level, but are no experts
> in tuning both.
> That's because the problems usually occur whenever lots of editors are
> working at the same time and we are having an increased editing load
> in the last weeks.

Are you seeing conflict errors in the event log?

----------
David Glick
Web Developer
[hidden email]
206.286.1235x32

Groundwire Consulting is here.

http://groundwire.org/about/FAQ-gw-consulting

------------------------------------------------------------------------------
Learn Windows Azure Live! Tuesday, Dec 13, 2011
Microsoft is holding a special Learn Windows Azure training event for
developers. It will provide a great way to learn Windows Azure and what it
provides. You can attend the event by watching it streamed LIVE online.
Learn more at http://p.sf.net/sfu/ms-windowsazure
_______________________________________________
Plone-Users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/plone-users

Nejc Zupan

Re: Performance problems

In reply to this post by Fabian Büchler

Hi Fabian,

you also might want to pin zope.i18nmessageid to 3.5.1. This helped a lot on our

Plone 3 sites -> http://plone.293351.n2.nabble.com/Severe-memory-leak-in-zope-i18nmessageid-fixed-td4917950.html

Regards,

Nejc Zupan

M: +386 (0)40 730 735
E: [hidden email]

www.niteoweb.com

On 16. dec. 2011, at 15:22, Fabian Büchler wrote:

Hello everyone,

our company develops / maintains the official tourism website for Vienna.

We are experiencing performance problems for quite some time now, but
increasing over the last weeks, and hope to find some help here.

We believe in some bottleneck at ZEO or ZODB level, but are no experts
in tuning both.
That's because the problems usually occur whenever lots of editors are
working at the same time and we are having an increased editing load
in the last weeks.

Can anyone provide us with some advice about zodb/zeo tuning
parameters or any good tools to get some useful metrics?
Also, if anyone knows (or is) experts who do (paid) consulting on that
topic, please feel free to refer them to me.

Thanks for any help in advance!
Regards, Fabian

Some insight into our system:

* It's a very complex Plone setup with lots of custom modifications to
Plone (used as a editing backend) and a custom frontend.
We are serving content for a couple of different subsites, an API and more.

* Currently we are still running Plone 3.2. (an upgrade to Plone 4.1
is in preparation, but won't be done before a few months from now)

* The installation is running on two dedicated servers (which are also
serving one smaller django installation and some static stuff), both
16 GB ram, 2x 2 GHz dualcore processors, powered by CentOS,...

* per month we have about 300.000 to 500.000 unique users, 3 to 12
mio. pageviews (GA vs. awstats), 1200 GB traffic,...

* the system setup is
* both servers run lighttpd > varnish > 3 zope instances / zeo clients
* one server runs zeo, and the zodb (currently about 2 GB, 10 GB
blobstorage) is on a DRBD shared storage

* server metrics say, that the servers are not overloaded. CPU usage
between 30% and 80% at peak times, I/O far from any critical numbers.

anything else you need to know?

------------------------------------------------------------------------------
Learn Windows Azure Live! Tuesday, Dec 13, 2011
Microsoft is holding a special Learn Windows Azure training event for
developers. It will provide a great way to learn Windows Azure and what it
provides. You can attend the event by watching it streamed LIVE online.
Learn more at http://p.sf.net/sfu/ms-windowsazure
_______________________________________________
Plone-Users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/plone-users

smime.p7s (2K) Download Attachment

ajung

Re: Performance problems

In reply to this post by Fabian Büchler

Some (perhaps unrelated) remarks:

- 42k objects in a catalog is barely nothing
- moving a catalog to a dedicated mountpoint may help but I doubt that it is suitable
for such a "small" catalog
- you may look into collective.stats for monitoring ZODB level activities (not sure how well it plays with ZEO)
- did you try reducing the number of worker threads per instance down to two or one?
- if you speak of heavy customizations: use ZopeProfiler or PTProfiler for profiling
the rendering of a typical "slow" page
- you must get an idea how many requests per second you receive in peak times and check how long processing takes (Zope's requestprofiler may help here)

-aj

Fabian Büchler wrote

We are planning to do some iterative changes to zeo client cache size
(from 100MB to 2/300 MB) and the zodb-cache-size (from 25000 to 100000
objects) and the zserver-threads (from not set to 8-10) in the next
weeks (always with some time to monitor the effects).
Do you have any advice on some good numbers there?

The portal_catalog says it holds 42,347 records. I don't know how many
objects the zodb holds in total. Do you know how to find out?
We've added only a few (~5) boolean values to the catalog, and 3-4
short strings, but nothing big.

With mounting the catalog to it's own moint point, you mean having a
second data.fs which holds just the catalog ?

Thanks for your quick answer!
Regards,
Fabian

2011/12/16 Nathan Van Gheem <[hidden email]>:
> Have you tweaked the zeo client cache settings zodb-cache-size option
> in http://pypi.python.org/pypi/plone.recipe.zope2instance/4.2 recipe?
>
> How large is your portal_catalog? Have you added a good deal of indexes and
> metadata? Mounting the catalog to it's own mount point helps out also.
>
> Take a look at http://scalingplone.pbworks.com/w/page/3679597/FrontPage for
> further tips.
>
> On Fri, Dec 16, 2011 at 8:22 AM, Fabian Büchler <[hidden email]>
> wrote:
>>
>> Hello everyone,
>>
>> our company develops / maintains the official tourism website for Vienna.
>>
>> We are experiencing performance problems for quite some time now, but
>> increasing over the last weeks, and hope to find some help here.
>>
>> We believe in some bottleneck at ZEO or ZODB level, but are no experts
>> in tuning both.
>> That's because the problems usually occur whenever lots of editors are
>> working at the same time and we are having an increased editing load
>> in the last weeks.
>>
>> Can anyone provide us with some advice about zodb/zeo tuning
>> parameters or any good tools to get some useful metrics?
>> Also, if anyone knows (or is) experts who do (paid) consulting on that
>> topic, please feel free to refer them to me.
>>
>> Thanks for any help in advance!
>> Regards, Fabian
>>
>>
>> Some insight into our system:
>>
>> * It's a very complex Plone setup with lots of custom modifications to
>> Plone (used as a editing backend) and a custom frontend.
>> We are serving content for a couple of different subsites, an API and
>> more.
>>
>> * Currently we are still running Plone 3.2. (an upgrade to Plone 4.1
>> is in preparation, but won't be done before a few months from now)
>>
>> * The installation is running on two dedicated servers (which are also
>> serving one smaller django installation and some static stuff), both
>> 16 GB ram, 2x 2 GHz dualcore processors, powered by CentOS,...
>>
>> * per month we have about 300.000 to 500.000 unique users, 3 to 12
>> mio. pageviews (GA vs. awstats), 1200 GB traffic,...
>>
>> * the system setup is
>> * both servers run lighttpd > varnish > 3 zope instances / zeo clients
>> * one server runs zeo, and the zodb (currently about 2 GB, 10 GB
>> blobstorage) is on a DRBD shared storage
>>
>> * server metrics say, that the servers are not overloaded. CPU usage
>> between 30% and 80% at peak times, I/O far from any critical numbers.
>>
>> anything else you need to know?
>>
>>
>> ------------------------------------------------------------------------------
>> Learn Windows Azure Live! Tuesday, Dec 13, 2011
>> Microsoft is holding a special Learn Windows Azure training event for
>> developers. It will provide a great way to learn Windows Azure and what it
>> provides. You can attend the event by watching it streamed LIVE online.
>> Learn more at http://p.sf.net/sfu/ms-windowsazure
>> _______________________________________________
>> Plone-Users mailing list
>> [hidden email]
>> https://lists.sourceforge.net/lists/listinfo/plone-users
>
>

------------------------------------------------------------------------------
Learn Windows Azure Live! Tuesday, Dec 13, 2011
Microsoft is holding a special Learn Windows Azure training event for
developers. It will provide a great way to learn Windows Azure and what it
provides. You can attend the event by watching it streamed LIVE online.
Learn more at http://p.sf.net/sfu/ms-windowsazure
_______________________________________________
Plone-Users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/plone-users

Alan Runyan-3

Re: Performance problems

In reply to this post by Fabian Büchler

We believe in some bottleneck at ZEO or ZODB level, but are no experts
in tuning both.
That's because the problems usually occur whenever lots of editors are
working at the same time and we are having an increased editing load
in the last weeks.

Can anyone provide us with some advice about zodb/zeo tuning
parameters or any good tools to get some useful metrics?
Also, if anyone knows (or is) experts who do (paid) consulting on that
topic, please feel free to refer them to me.

Isolate editors to a particular load balancing pool. The editor pool should be

different from the public/anonymous pool.

Use the collective.stats package.

--
Alan Runyan
Office +17135305244 | Skype/Twitter @runyaga
President of Enfold Systems | http://www.enfoldsystems.com/
Co-founder of the Plone CMS | Free Plone site in less than 10 seconds - http://ploud.com/

------------------------------------------------------------------------------
Learn Windows Azure Live! Tuesday, Dec 13, 2011
Microsoft is holding a special Learn Windows Azure training event for
developers. It will provide a great way to learn Windows Azure and what it
provides. You can attend the event by watching it streamed LIVE online.
Learn more at http://p.sf.net/sfu/ms-windowsazure
_______________________________________________
Plone-Users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/plone-users

Max Jakob

Re: Performance problems

In reply to this post by Nathan Van Gheem

On 12/16/2011 04:46 PM, Nathan Van Gheem wrote:

> zodb-cache-size (from 25000 to 100000
>
> Cache as much as you can.
>
> zserver-threads (from not set to 8-10)
>
> This might be your problem. The more you have, the slower performance
> you get. Only use 1 or 2 and use more clients if you need better
> performance.
>

Hi,
I got a similar performance problem.
In my setup Plone runs on a machine with 8 cores. There are 8 zeoclients
running and I increased the zserver-threads to 8.
Does it make sense to use let's say 30 (or even more) zeoclients in
this case (More then 1 client per core?) and decrease zserver-threads to
2 (or 1)?

\Max

>
> On Fri, Dec 16, 2011 at 9:33 AM, Fabian Büchler
> <[hidden email]
> <mailto:[hidden email]>> wrote:
>
> We are planning to do some iterative changes to zeo client cache size
> (from 100MB to 2/300 MB) and the zodb-cache-size (from 25000 to 100000
> objects) and the zserver-threads (from not set to 8-10) in the next
> weeks (always with some time to monitor the effects).
> Do you have any advice on some good numbers there?
>
> The portal_catalog says it holds 42,347 records. I don't know how many
> objects the zodb holds in total. Do you know how to find out?
> We've added only a few (~5) boolean values to the catalog, and 3-4
> short strings, but nothing big.
>
> With mounting the catalog to it's own moint point, you mean having a
> second data.fs which holds just the catalog ?
>
> Thanks for your quick answer!
> Regards,
> Fabian
>
> 2011/12/16 Nathan Van Gheem
> <[hidden email]
> <mailto:[hidden email]>>:
> > Have you tweaked the zeo client cache settings zodb-cache-size option
> > in http://pypi.python.org/pypi/plone.recipe.zope2instance/4.2 recipe?
> >
> > How large is your portal_catalog? Have you added a good deal of
> indexes and
> > metadata? Mounting the catalog to it's own mount point helps out
> also.
> >
> > Take a look at
> http://scalingplone.pbworks.com/w/page/3679597/FrontPage for
> > further tips.
> >
> > On Fri, Dec 16, 2011 at 8:22 AM, Fabian Büchler
> <[hidden email]
> <mailto:[hidden email]>>
> > wrote:
> >>
> >> Hello everyone,
> >>
> >> our company develops / maintains the official tourism website
> for Vienna.
> >>
> >> We are experiencing performance problems for quite some time
> now, but
> >> increasing over the last weeks, and hope to find some help here.
> >>
> >> We believe in some bottleneck at ZEO or ZODB level, but are no
> experts
> >> in tuning both.
> >> That's because the problems usually occur whenever lots of
> editors are
> >> working at the same time and we are having an increased editing load
> >> in the last weeks.
> >>
> >> Can anyone provide us with some advice about zodb/zeo tuning
> >> parameters or any good tools to get some useful metrics?
> >> Also, if anyone knows (or is) experts who do (paid) consulting
> on that
> >> topic, please feel free to refer them to me.
> >>
> >> Thanks for any help in advance!
> >> Regards, Fabian
> >>
> >>
> >> Some insight into our system:
> >>
> >> * It's a very complex Plone setup with lots of custom
> modifications to
> >> Plone (used as a editing backend) and a custom frontend.
> >> We are serving content for a couple of different subsites, an
> API and
> >> more.
> >>
> >> * Currently we are still running Plone 3.2. (an upgrade to Plone 4.1
> >> is in preparation, but won't be done before a few months from now)
> >>
> >> * The installation is running on two dedicated servers (which
> are also
> >> serving one smaller django installation and some static stuff), both
> >> 16 GB ram, 2x 2 GHz dualcore processors, powered by CentOS,...
> >>
> >> * per month we have about 300.000 to 500.000 unique users, 3 to 12
> >> mio. pageviews (GA vs. awstats), 1200 GB traffic,...
> >>
> >> * the system setup is
> >> * both servers run lighttpd > varnish > 3 zope instances / zeo
> clients
> >> * one server runs zeo, and the zodb (currently about 2 GB, 10 GB
> >> blobstorage) is on a DRBD shared storage
> >>
> >> * server metrics say, that the servers are not overloaded. CPU usage
> >> between 30% and 80% at peak times, I/O far from any critical
> numbers.
> >>
> >> anything else you need to know?
> >>
> >>
> >>
> ------------------------------------------------------------------------------
> >> Learn Windows Azure Live! Tuesday, Dec 13, 2011
> >> Microsoft is holding a special Learn Windows Azure training
> event for
> >> developers. It will provide a great way to learn Windows Azure
> and what it
> >> provides. You can attend the event by watching it streamed LIVE
> online.
> >> Learn more at http://p.sf.net/sfu/ms-windowsazure
> >> _______________________________________________
> >> Plone-Users mailing list
> >> [hidden email]
> <mailto:[hidden email]>
> >> https://lists.sourceforge.net/lists/listinfo/plone-users
> >
> >
>
>
>
>
> ------------------------------------------------------------------------------
> Learn Windows Azure Live! Tuesday, Dec 13, 2011
> Microsoft is holding a special Learn Windows Azure training event for
> developers. It will provide a great way to learn Windows Azure and what it
> provides. You can attend the event by watching it streamed LIVE online.
> Learn more at http://p.sf.net/sfu/ms-windowsazure
>
>
>
> _______________________________________________
> Plone-Users mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/plone-users

ajung

Re: Performance problems

Max Jakob wrote

On 12/16/2011 04:46 PM, Nathan Van Gheem wrote:
> zodb-cache-size (from 25000 to 100000
>
> Cache as much as you can.
>
> zserver-threads (from not set to 8-10)
>
> This might be your problem. The more you have, the slower performance
> you get. Only use 1 or 2 and use more clients if you need better
> performance.
>

There seems to be consensus that more ZEO clients with less worker threads is better than less ZEO clients with more worker threads.

I second Alans advice having dedicated instances for editors and for the public (anonymous) part of the site...but YMMV depending on the structure of the site and its use (large corporate sites like university sites with possibly "lots" of editors and lots of writes...each case is different :->).

-aj

Fabian Büchler

Re: Performance problems

In reply to this post by Nathan Van Gheem

Thanks Nathan!
I'll try the caching parameters first, and add more zeo clients later,
if thtat is still necessary.

Regards, Fabian

2011/12/16 Nathan Van Gheem <[hidden email]>:

>> zodb-cache-size (from 25000 to 100000
>
> Cache as much as you can.
>
>> zserver-threads (from not set to 8-10)
>
> This might be your problem. The more you have, the slower performance you
> get. Only use 1 or 2 and use more clients if you need better performance.
>
>
> On Fri, Dec 16, 2011 at 9:33 AM, Fabian Büchler <[hidden email]>
> wrote:
>>
>> We are planning to do some iterative changes to zeo client cache size
>> (from 100MB to 2/300 MB) and the zodb-cache-size (from 25000 to 100000
>> objects) and the zserver-threads (from not set to 8-10) in the next
>> weeks (always with some time to monitor the effects).
>> Do you have any advice on some good numbers there?
>>
>> The portal_catalog says it holds 42,347 records. I don't know how many
>> objects the zodb holds in total. Do you know how to find out?
>> We've added only a few (~5) boolean values to the catalog, and 3-4
>> short strings, but nothing big.
>>
>> With mounting the catalog to it's own moint point, you mean having a
>> second data.fs which holds just the catalog ?
>>
>> Thanks for your quick answer!
>> Regards,
>> Fabian
>>
>> 2011/12/16 Nathan Van Gheem <[hidden email]>:
>> > Have you tweaked the zeo client cache settings zodb-cache-size option
>> > in http://pypi.python.org/pypi/plone.recipe.zope2instance/4.2 recipe?
>> >
>> > How large is your portal_catalog? Have you added a good deal of indexes
>> > and
>> > metadata? Mounting the catalog to it's own mount point helps out also.
>> >
>> > Take a look at http://scalingplone.pbworks.com/w/page/3679597/FrontPage
>> > for
>> > further tips.
>> >
>> > On Fri, Dec 16, 2011 at 8:22 AM, Fabian Büchler
>> > <[hidden email]>
>> > wrote:
>> >>
>> >> Hello everyone,
>> >>
>> >> our company develops / maintains the official tourism website for
>> >> Vienna.
>> >>
>> >> We are experiencing performance problems for quite some time now, but
>> >> increasing over the last weeks, and hope to find some help here.
>> >>
>> >> We believe in some bottleneck at ZEO or ZODB level, but are no experts
>> >> in tuning both.
>> >> That's because the problems usually occur whenever lots of editors are
>> >> working at the same time and we are having an increased editing load
>> >> in the last weeks.
>> >>
>> >> Can anyone provide us with some advice about zodb/zeo tuning
>> >> parameters or any good tools to get some useful metrics?
>> >> Also, if anyone knows (or is) experts who do (paid) consulting on that
>> >> topic, please feel free to refer them to me.
>> >>
>> >> Thanks for any help in advance!
>> >> Regards, Fabian
>> >>
>> >>
>> >> Some insight into our system:
>> >>
>> >> * It's a very complex Plone setup with lots of custom modifications to
>> >> Plone (used as a editing backend) and a custom frontend.
>> >> We are serving content for a couple of different subsites, an API and
>> >> more.
>> >>
>> >> * Currently we are still running Plone 3.2. (an upgrade to Plone 4.1
>> >> is in preparation, but won't be done before a few months from now)
>> >>
>> >> * The installation is running on two dedicated servers (which are also
>> >> serving one smaller django installation and some static stuff), both
>> >> 16 GB ram, 2x 2 GHz dualcore processors, powered by CentOS,...
>> >>
>> >> * per month we have about 300.000 to 500.000 unique users, 3 to 12
>> >> mio. pageviews (GA vs. awstats), 1200 GB traffic,...
>> >>
>> >> * the system setup is
>> >> * both servers run lighttpd > varnish > 3 zope instances / zeo clients
>> >> * one server runs zeo, and the zodb (currently about 2 GB, 10 GB
>> >> blobstorage) is on a DRBD shared storage
>> >>
>> >> * server metrics say, that the servers are not overloaded. CPU usage
>> >> between 30% and 80% at peak times, I/O far from any critical numbers.
>> >>
>> >> anything else you need to know?
>> >>
>> >>
>> >>
>> >> ------------------------------------------------------------------------------
>> >> Learn Windows Azure Live! Tuesday, Dec 13, 2011
>> >> Microsoft is holding a special Learn Windows Azure training event for
>> >> developers. It will provide a great way to learn Windows Azure and what
>> >> it
>> >> provides. You can attend the event by watching it streamed LIVE online.
>> >> Learn more at http://p.sf.net/sfu/ms-windowsazure
>> >> _______________________________________________
>> >> Plone-Users mailing list
>> >> [hidden email]
>> >> https://lists.sourceforge.net/lists/listinfo/plone-users
>> >
>> >
>
>

Nathan Van Gheem

Re: Performance problems

For everyone running plone sites, http://scalingplone.pbworks.com/w/page/3770062/Tuning is a must read. You should never use 8 zserver-threads.

Also, consider this, Plone 4 is considerably faster and you can increase performance even more by using the Chameleon template engine.

On Sat, Dec 17, 2011 at 7:55 AM, Fabian Büchler <[hidden email]> wrote:

Thanks Nathan!
I'll try the caching parameters first, and add more zeo clients later,
if thtat is still necessary.

Regards, Fabian

2011/12/16 Nathan Van Gheem <[hidden email]>:
>> zodb-cache-size (from 25000 to 100000
>
> Cache as much as you can.
>
>> zserver-threads (from not set to 8-10)
>
> This might be your problem. The more you have, the slower performance you
> get. Only use 1 or 2 and use more clients if you need better performance.
>
>
> On Fri, Dec 16, 2011 at 9:33 AM, Fabian Büchler <[hidden email]>
> wrote:
>>
>> We are planning to do some iterative changes to zeo client cache size
>> (from 100MB to 2/300 MB) and the zodb-cache-size (from 25000 to 100000
>> objects) and the zserver-threads (from not set to 8-10) in the next
>> weeks (always with some time to monitor the effects).
>> Do you have any advice on some good numbers there?
>>
>> The portal_catalog says it holds 42,347 records. I don't know how many
>> objects the zodb holds in total. Do you know how to find out?
>> We've added only a few (~5) boolean values to the catalog, and 3-4
>> short strings, but nothing big.
>>
>> With mounting the catalog to it's own moint point, you mean having a
>> second data.fs which holds just the catalog ?
>>
>> Thanks for your quick answer!
>> Regards,
>> Fabian
>>
>> 2011/12/16 Nathan Van Gheem <[hidden email]>:
>> > Have you tweaked the zeo client cache settings zodb-cache-size option
>> > in http://pypi.python.org/pypi/plone.recipe.zope2instance/4.2 recipe?
>> >
>> > How large is your portal_catalog? Have you added a good deal of indexes
>> > and
>> > metadata? Mounting the catalog to it's own mount point helps out also.
>> >
>> > Take a look at http://scalingplone.pbworks.com/w/page/3679597/FrontPage
>> > for
>> > further tips.
>> >
>> > On Fri, Dec 16, 2011 at 8:22 AM, Fabian Büchler
>> > <[hidden email]>
>> > wrote:
>> >>
>> >> Hello everyone,
>> >>
>> >> our company develops / maintains the official tourism website for
>> >> Vienna.
>> >>
>> >> We are experiencing performance problems for quite some time now, but
>> >> increasing over the last weeks, and hope to find some help here.
>> >>
>> >> We believe in some bottleneck at ZEO or ZODB level, but are no experts
>> >> in tuning both.
>> >> That's because the problems usually occur whenever lots of editors are
>> >> working at the same time and we are having an increased editing load
>> >> in the last weeks.
>> >>
>> >> Can anyone provide us with some advice about zodb/zeo tuning
>> >> parameters or any good tools to get some useful metrics?
>> >> Also, if anyone knows (or is) experts who do (paid) consulting on that
>> >> topic, please feel free to refer them to me.
>> >>
>> >> Thanks for any help in advance!
>> >> Regards, Fabian
>> >>
>> >>
>> >> Some insight into our system:
>> >>
>> >> * It's a very complex Plone setup with lots of custom modifications to
>> >> Plone (used as a editing backend) and a custom frontend.
>> >> We are serving content for a couple of different subsites, an API and
>> >> more.
>> >>
>> >> * Currently we are still running Plone 3.2. (an upgrade to Plone 4.1
>> >> is in preparation, but won't be done before a few months from now)
>> >>
>> >> * The installation is running on two dedicated servers (which are also
>> >> serving one smaller django installation and some static stuff), both
>> >> 16 GB ram, 2x 2 GHz dualcore processors, powered by CentOS,...
>> >>
>> >> * per month we have about 300.000 to 500.000 unique users, 3 to 12
>> >> mio. pageviews (GA vs. awstats), 1200 GB traffic,...
>> >>
>> >> * the system setup is
>> >> * both servers run lighttpd > varnish > 3 zope instances / zeo clients
>> >> * one server runs zeo, and the zodb (currently about 2 GB, 10 GB
>> >> blobstorage) is on a DRBD shared storage
>> >>
>> >> * server metrics say, that the servers are not overloaded. CPU usage
>> >> between 30% and 80% at peak times, I/O far from any critical numbers.
>> >>
>> >> anything else you need to know?
>> >>
>> >>
>> >>
>> >> ------------------------------------------------------------------------------
>> >> Learn Windows Azure Live! Tuesday, Dec 13, 2011
>> >> Microsoft is holding a special Learn Windows Azure training event for
>> >> developers. It will provide a great way to learn Windows Azure and what
>> >> it
>> >> provides. You can attend the event by watching it streamed LIVE online.
>> >> Learn more at http://p.sf.net/sfu/ms-windowsazure
>> >> _______________________________________________
>> >> Plone-Users mailing list
>> >> [hidden email]
>> >> https://lists.sourceforge.net/lists/listinfo/plone-users
>> >
>> >
>
>

Fabian Büchler

Re: Performance problems

In reply to this post by Nejc Zupan

Hi Nejc,

On 16.12.2011, at 18:21, Nejc Zupan wrote:

Hi Fabian,

you also might want to pin zope.i18nmessageid to 3.5.1. This helped a lot on our
Plone 3 sites -> http://plone.293351.n2.nabble.com/Severe-memory-leak-in-zope-i18nmessageid-fixed-td4917950.html

wow, thanks for that!

We always had some kind of memory leaking problems on the system.

Was worse some time ago and stabilized a bit in the last months, but that might help to finally get it fixed.

Regards,
Nejc Zupan

Regards,

Fabian

M: +386 (0)40 730 735
E: [hidden email]

www.niteoweb.com

On 16. dec. 2011, at 15:22, Fabian Büchler wrote:

Hello everyone,

our company develops / maintains the official tourism website for Vienna.

We are experiencing performance problems for quite some time now, but
increasing over the last weeks, and hope to find some help here.

We believe in some bottleneck at ZEO or ZODB level, but are no experts
in tuning both.
That's because the problems usually occur whenever lots of editors are
working at the same time and we are having an increased editing load
in the last weeks.

Can anyone provide us with some advice about zodb/zeo tuning
parameters or any good tools to get some useful metrics?
Also, if anyone knows (or is) experts who do (paid) consulting on that
topic, please feel free to refer them to me.

Thanks for any help in advance!
Regards, Fabian

Some insight into our system:

* It's a very complex Plone setup with lots of custom modifications to
Plone (used as a editing backend) and a custom frontend.
We are serving content for a couple of different subsites, an API and more.

* Currently we are still running Plone 3.2. (an upgrade to Plone 4.1
is in preparation, but won't be done before a few months from now)

* The installation is running on two dedicated servers (which are also
serving one smaller django installation and some static stuff), both
16 GB ram, 2x 2 GHz dualcore processors, powered by CentOS,...

* per month we have about 300.000 to 500.000 unique users, 3 to 12
mio. pageviews (GA vs. awstats), 1200 GB traffic,...

* the system setup is
* both servers run lighttpd > varnish > 3 zope instances / zeo clients
* one server runs zeo, and the zodb (currently about 2 GB, 10 GB
blobstorage) is on a DRBD shared storage

* server metrics say, that the servers are not overloaded. CPU usage
between 30% and 80% at peak times, I/O far from any critical numbers.

anything else you need to know?

------------------------------------------------------------------------------
Learn Windows Azure Live! Tuesday, Dec 13, 2011
Microsoft is holding a special Learn Windows Azure training event for
developers. It will provide a great way to learn Windows Azure and what it
provides. You can attend the event by watching it streamed LIVE online.
Learn more at http://p.sf.net/sfu/ms-windowsazure
_______________________________________________
Plone-Users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/plone-users

------------------------------------------------------------------------------
Learn Windows Azure Live! Tuesday, Dec 13, 2011
Microsoft is holding a special Learn Windows Azure training event for
developers. It will provide a great way to learn Windows Azure and what it
provides. You can attend the event by watching it streamed LIVE online.
Learn more at http://p.sf.net/sfu/ms-windowsazure_______________________________________________
Plone-Users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/plone-users

Nathan Van Gheem

Re: Performance problems

Python 2.4 has issues with memory management so usually what people do who run plone 3 is restart the clients nightly or running a cron job every few minutes that checks the RAM usage of each client and restarts it if it's over a certain threshold.

2011/12/17 Fabian Büchler <[hidden email]>

Hi Nejc,

On 16.12.2011, at 18:21, Nejc Zupan wrote:

Hi Fabian,

you also might want to pin zope.i18nmessageid to 3.5.1. This helped a lot on our
Plone 3 sites -> http://plone.293351.n2.nabble.com/Severe-memory-leak-in-zope-i18nmessageid-fixed-td4917950.html

wow, thanks for that!
We always had some kind of memory leaking problems on the system.
Was worse some time ago and stabilized a bit in the last months, but that might help to finally get it fixed.

Regards,
Nejc Zupan

Regards,
Fabian

M: <a href="tel:%2B386%20%280%2940%20730%20735" value="+38640730735" target="_blank">+386 (0)40 730 735
E: [hidden email]

www.niteoweb.com

On 16. dec. 2011, at 15:22, Fabian Büchler wrote:

Hello everyone,

our company develops / maintains the official tourism website for Vienna.

We are experiencing performance problems for quite some time now, but
increasing over the last weeks, and hope to find some help here.

We believe in some bottleneck at ZEO or ZODB level, but are no experts
in tuning both.
That's because the problems usually occur whenever lots of editors are
working at the same time and we are having an increased editing load
in the last weeks.

Can anyone provide us with some advice about zodb/zeo tuning
parameters or any good tools to get some useful metrics?
Also, if anyone knows (or is) experts who do (paid) consulting on that
topic, please feel free to refer them to me.

Thanks for any help in advance!
Regards, Fabian

Some insight into our system:

* It's a very complex Plone setup with lots of custom modifications to
Plone (used as a editing backend) and a custom frontend.
We are serving content for a couple of different subsites, an API and more.

* Currently we are still running Plone 3.2. (an upgrade to Plone 4.1
is in preparation, but won't be done before a few months from now)

* The installation is running on two dedicated servers (which are also
serving one smaller django installation and some static stuff), both
16 GB ram, 2x 2 GHz dualcore processors, powered by CentOS,...

* per month we have about 300.000 to 500.000 unique users, 3 to 12
mio. pageviews (GA vs. awstats), 1200 GB traffic,...

* the system setup is
* both servers run lighttpd > varnish > 3 zope instances / zeo clients
* one server runs zeo, and the zodb (currently about 2 GB, 10 GB
blobstorage) is on a DRBD shared storage

* server metrics say, that the servers are not overloaded. CPU usage
between 30% and 80% at peak times, I/O far from any critical numbers.

anything else you need to know?

------------------------------------------------------------------------------
Learn Windows Azure Live! Tuesday, Dec 13, 2011
Microsoft is holding a special Learn Windows Azure training event for
developers. It will provide a great way to learn Windows Azure and what it
provides. You can attend the event by watching it streamed LIVE online.
Learn more at http://p.sf.net/sfu/ms-windowsazure
_______________________________________________
Plone-Users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/plone-users

------------------------------------------------------------------------------
Learn Windows Azure Live! Tuesday, Dec 13, 2011
Microsoft is holding a special Learn Windows Azure training event for
developers. It will provide a great way to learn Windows Azure and what it
provides. You can attend the event by watching it streamed LIVE online.
Learn more at http://p.sf.net/sfu/ms-windowsazure_______________________________________________
Plone-Users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/plone-users

------------------------------------------------------------------------------
Learn Windows Azure Live! Tuesday, Dec 13, 2011
Microsoft is holding a special Learn Windows Azure training event for
developers. It will provide a great way to learn Windows Azure and what it
provides. You can attend the event by watching it streamed LIVE online.
Learn more at http://p.sf.net/sfu/ms-windowsazure
_______________________________________________
Plone-Users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/plone-users

Nathan Van Gheem

Re: Performance problems

BTW, another reason to move to Plone 4...

On Sat, Dec 17, 2011 at 8:31 AM, Nathan Van Gheem <[hidden email]> wrote:

Python 2.4 has issues with memory management so usually what people do who run plone 3 is restart the clients nightly or running a cron job every few minutes that checks the RAM usage of each client and restarts it if it's over a certain threshold.

2011/12/17 Fabian Büchler <[hidden email]>

Hi Nejc,

On 16.12.2011, at 18:21, Nejc Zupan wrote:

Hi Fabian,

you also might want to pin zope.i18nmessageid to 3.5.1. This helped a lot on our
Plone 3 sites -> http://plone.293351.n2.nabble.com/Severe-memory-leak-in-zope-i18nmessageid-fixed-td4917950.html

wow, thanks for that!
We always had some kind of memory leaking problems on the system.
Was worse some time ago and stabilized a bit in the last months, but that might help to finally get it fixed.

Regards,
Nejc Zupan

Regards,
Fabian

M: <a href="tel:%2B386%20%280%2940%20730%20735" value="+38640730735" target="_blank">+386 (0)40 730 735
E: [hidden email]

www.niteoweb.com

On 16. dec. 2011, at 15:22, Fabian Büchler wrote:

Hello everyone,

our company develops / maintains the official tourism website for Vienna.

We are experiencing performance problems for quite some time now, but
increasing over the last weeks, and hope to find some help here.

We believe in some bottleneck at ZEO or ZODB level, but are no experts
in tuning both.
That's because the problems usually occur whenever lots of editors are
working at the same time and we are having an increased editing load
in the last weeks.

Can anyone provide us with some advice about zodb/zeo tuning
parameters or any good tools to get some useful metrics?
Also, if anyone knows (or is) experts who do (paid) consulting on that
topic, please feel free to refer them to me.

Thanks for any help in advance!
Regards, Fabian

Some insight into our system:

* It's a very complex Plone setup with lots of custom modifications to
Plone (used as a editing backend) and a custom frontend.
We are serving content for a couple of different subsites, an API and more.

* Currently we are still running Plone 3.2. (an upgrade to Plone 4.1
is in preparation, but won't be done before a few months from now)

* The installation is running on two dedicated servers (which are also
serving one smaller django installation and some static stuff), both
16 GB ram, 2x 2 GHz dualcore processors, powered by CentOS,...

* per month we have about 300.000 to 500.000 unique users, 3 to 12
mio. pageviews (GA vs. awstats), 1200 GB traffic,...

* the system setup is
* both servers run lighttpd > varnish > 3 zope instances / zeo clients
* one server runs zeo, and the zodb (currently about 2 GB, 10 GB
blobstorage) is on a DRBD shared storage

* server metrics say, that the servers are not overloaded. CPU usage
between 30% and 80% at peak times, I/O far from any critical numbers.

anything else you need to know?

------------------------------------------------------------------------------
Learn Windows Azure Live! Tuesday, Dec 13, 2011
Microsoft is holding a special Learn Windows Azure training event for
developers. It will provide a great way to learn Windows Azure and what it
provides. You can attend the event by watching it streamed LIVE online.
Learn more at http://p.sf.net/sfu/ms-windowsazure
_______________________________________________
Plone-Users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/plone-users

------------------------------------------------------------------------------
Learn Windows Azure Live! Tuesday, Dec 13, 2011
Microsoft is holding a special Learn Windows Azure training event for
developers. It will provide a great way to learn Windows Azure and what it
provides. You can attend the event by watching it streamed LIVE online.
Learn more at http://p.sf.net/sfu/ms-windowsazure_______________________________________________
Plone-Users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/plone-users

------------------------------------------------------------------------------
Learn Windows Azure Live! Tuesday, Dec 13, 2011
Microsoft is holding a special Learn Windows Azure training event for
developers. It will provide a great way to learn Windows Azure and what it
provides. You can attend the event by watching it streamed LIVE online.
Learn more at http://p.sf.net/sfu/ms-windowsazure
_______________________________________________
Plone-Users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/plone-users

Fabian Büchler

Re: Performance problems

In reply to this post by ajung

On 16.12.2011, at 19:34, ajung wrote:

> Some (perhaps unrelated) remarks:
>
> - 42k objects in a catalog is barely nothing
> - moving a catalog to a dedicated mountpoint may help but I doubt that it is
> suitable
> for such a "small" catalog

alright - I also doubted, that our catalog is really a big one.
I'll keep this as a last option, if the other tips don't work out.

> - you may look into collective.stats for monitoring ZODB level activities
> (not sure how well it plays with ZEO)

> - did you try reducing the number of worker threads per instance down to two
> or one?

Not yet, but I'm planning to do this soon.
Wanted to get some advice on what could help first. As I see it now: more clients, less threads and more cache size.

> - if you speak of heavy customizations: use ZopeProfiler or PTProfiler for
> profiling
> the rendering of a typical "slow" page

We've already done that regularly when problems arise.
In that case it's really about overall performance - not just for a specific view.
Just think we passed some threshold of how many requests our system could handle with the current cache/threads/clients configuration.

> - you must get an idea how many requests per second you receive in peak
> times and check how long processing takes (Zope's requestprofiler may help
> here)

Thanks for the tips! I'll look into request profiler, too.

> -aj
>

Regards, Fabian

>
>
> Fabian Büchler wrote
>>
>> We are planning to do some iterative changes to zeo client cache size
>> (from 100MB to 2/300 MB) and the zodb-cache-size (from 25000 to 100000
>> objects) and the zserver-threads (from not set to 8-10) in the next
>> weeks (always with some time to monitor the effects).
>> Do you have any advice on some good numbers there?
>>
>> The portal_catalog says it holds 42,347 records. I don't know how many
>> objects the zodb holds in total. Do you know how to find out?
>> We've added only a few (~5) boolean values to the catalog, and 3-4
>> short strings, but nothing big.
>>
>> With mounting the catalog to it's own moint point, you mean having a
>> second data.fs which holds just the catalog ?
>>
>> Thanks for your quick answer!
>> Regards,
>> Fabian
>>
>> 2011/12/16 Nathan Van Gheem <vangheem@>:
>>> Have you tweaked the zeo client cache settings zodb-cache-size option
>>> in http://pypi.python.org/pypi/plone.recipe.zope2instance/4.2 recipe?
>>>
>>> How large is your portal_catalog? Have you added a good deal of indexes
>>> and
>>> metadata? Mounting the catalog to it's own mount point helps out also.
>>>
>>> Take a look at http://scalingplone.pbworks.com/w/page/3679597/FrontPage
>>> for
>>> further tips.
>>>
>>> On Fri, Dec 16, 2011 at 8:22 AM, Fabian Büchler <fabian.buechler@>
>>> wrote:
>>>>
>>>> Hello everyone,
>>>>
>>>> our company develops / maintains the official tourism website for
>>>> Vienna.
>>>>
>>>> We are experiencing performance problems for quite some time now, but
>>>> increasing over the last weeks, and hope to find some help here.
>>>>
>>>> We believe in some bottleneck at ZEO or ZODB level, but are no experts
>>>> in tuning both.
>>>> That's because the problems usually occur whenever lots of editors are
>>>> working at the same time and we are having an increased editing load
>>>> in the last weeks.
>>>>
>>>> Can anyone provide us with some advice about zodb/zeo tuning
>>>> parameters or any good tools to get some useful metrics?
>>>> Also, if anyone knows (or is) experts who do (paid) consulting on that
>>>> topic, please feel free to refer them to me.
>>>>
>>>> Thanks for any help in advance!
>>>> Regards, Fabian
>>>>
>>>>
>>>> Some insight into our system:
>>>>
>>>> * It's a very complex Plone setup with lots of custom modifications to
>>>> Plone (used as a editing backend) and a custom frontend.
>>>> We are serving content for a couple of different subsites, an API and
>>>> more.
>>>>
>>>> * Currently we are still running Plone 3.2. (an upgrade to Plone 4.1
>>>> is in preparation, but won't be done before a few months from now)
>>>>
>>>> * The installation is running on two dedicated servers (which are also
>>>> serving one smaller django installation and some static stuff), both
>>>> 16 GB ram, 2x 2 GHz dualcore processors, powered by CentOS,...
>>>>
>>>> * per month we have about 300.000 to 500.000 unique users, 3 to 12
>>>> mio. pageviews (GA vs. awstats), 1200 GB traffic,...
>>>>
>>>> * the system setup is
>>>> * both servers run lighttpd > varnish > 3 zope instances / zeo clients
>>>> * one server runs zeo, and the zodb (currently about 2 GB, 10 GB
>>>> blobstorage) is on a DRBD shared storage
>>>>
>>>> * server metrics say, that the servers are not overloaded. CPU usage
>>>> between 30% and 80% at peak times, I/O far from any critical numbers.
>>>>
>>>> anything else you need to know?
>>>>
>>>>
>>>> ------------------------------------------------------------------------------
>>>> Learn Windows Azure Live! Tuesday, Dec 13, 2011
>>>> Microsoft is holding a special Learn Windows Azure training event for
>>>> developers. It will provide a great way to learn Windows Azure and what
>>>> it
>>>> provides. You can attend the event by watching it streamed LIVE online.
>>>> Learn more at http://p.sf.net/sfu/ms-windowsazure
>>>> _______________________________________________
>>>> Plone-Users mailing list
>>>> Plone-Users@.sourceforge
>>>> https://lists.sourceforge.net/lists/listinfo/plone-users
>>>
>>>
>>
>> ------------------------------------------------------------------------------
>> Learn Windows Azure Live! Tuesday, Dec 13, 2011
>> Microsoft is holding a special Learn Windows Azure training event for
>> developers. It will provide a great way to learn Windows Azure and what it
>> provides. You can attend the event by watching it streamed LIVE online.
>> Learn more at http://p.sf.net/sfu/ms-windowsazure
>> _______________________________________________
>> Plone-Users mailing list
>> Plone-Users@.sourceforge
>> https://lists.sourceforge.net/lists/listinfo/plone-users
>>
>
>
> --
> View this message in context: http://plone.293351.n2.nabble.com/Performance-problems-tp7101195p7102007.html
> Sent from the General Questions mailing list archive at Nabble.com.
>
> ------------------------------------------------------------------------------
> Learn Windows Azure Live! Tuesday, Dec 13, 2011
> Microsoft is holding a special Learn Windows Azure training event for
> developers. It will provide a great way to learn Windows Azure and what it
> provides. You can attend the event by watching it streamed LIVE online.
> Learn more at http://p.sf.net/sfu/ms-windowsazure
> _______________________________________________
> Plone-Users mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/plone-users

ajung

Re: Performance problems

Fabian Büchler wrote

> - if you speak of heavy customizations: use ZopeProfiler or PTProfiler for
> profiling
> the rendering of a typical "slow" page

We've already done that regularly when problems arise.
In that case it's really about overall performance - not just for a specific view.
Just think we passed some threshold of how many requests our system could handle with the current cache/threads/clients configuration.

Doing this regularly is a good thing. It often happens at even simple changes or new code can have a bad impact - either due to obvious programming errors or through some implicit implications one can not foresee (just because too much magic is happening under the hood). One common pitfall often seen is using tal:condition="image" for checking if there is actually an image....the code looks at the first glance perfect but the impact with large images can be horrible.

-aj

Matt Hamilton

Re: Performance problems

In reply to this post by Fabian Büchler

Fabian Büchler <fabian.buechler@...> writes:

> We believe in some bottleneck at ZEO or ZODB level, but are no experts
> in tuning both.

Some things not yet mentioned off the top of my head:

1) Install:

- experimental.aggressiveopaquespeedup
This helps with the traversal through the site. In Plone 3 if you traverse *through* a
folder with a large number of objects you wake up *all* objects in that folder. Typically
this occurs when you have something like a 'news' folder and you select 5 items for the
homepage. You actually load *all* items from the ZODB

- experimental.catalogqueryplan (you will need to pin to the last Plone 3.x
version which IIRC is 3.0.2

- experimental.contentcreation
This speeds up the act of initially creating new content (eg clicking
'page' from the add new menu). It puts in a 'dummy' catalog in portal_factory which
prevents the object being indexed/unindexed as it is created

- experimental.daterangeindexoptimisations
You will need to re-index your catalog after this to take effect, but it optimises the
date indexes for you.

2) Run the catalog optimise script:
https://github.com/hannosch/scripts/commits/master/catalogoptimize.py
This will optimise the BTree buckets in your catalog

3) Install collective.stats
This will tell you if you 'reads' are actually doing 'writes' to your database and how many
objects loaded and stored for each request.

-Matt

--
Matt Hamilton [hidden email]
Netsight Internet Solutions, Ltd. Business Vision on the Internet
http://www.netsight.co.uk +44 (0)117 9090901
Web Design | Zope/Plone Development and Consulting | Co-location | Hosting

------------------------------------------------------------------------------
Learn Windows Azure Live! Tuesday, Dec 13, 2011
Microsoft is holding a special Learn Windows Azure training event for
developers. It will provide a great way to learn Windows Azure and what it
provides. You can attend the event by watching it streamed LIVE online.
Learn more at http://p.sf.net/sfu/ms-windowsazure
_______________________________________________
Plone-Users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/plone-users

Fabian Büchler

Re: Performance problems

Thanks for your tips, Matt.
We'll try to go with the "cheap" (not a lot of work) modifications
first and try other optimizations later on, if we still have problems.
Anyway we'll upgrade to Plone4 early next year and I believe that will
help a lot anyway.

Thanks again everybody for all the great help!
Regards, Fabian

2011/12/19 Matt Hamilton <[hidden email]>:

> Fabian Büchler <fabian.buechler@...> writes:
>
>> We believe in some bottleneck at ZEO or ZODB level, but are no experts
>> in tuning both.
>
> Some things not yet mentioned off the top of my head:
>
> 1) Install:
>
> - experimental.aggressiveopaquespeedup
> This helps with the traversal through the site. In Plone 3 if you traverse *through* a
> folder with a large number of objects you wake up *all* objects in that folder. Typically
> this occurs when you have something like a 'news' folder and you select 5 items for the
> homepage. You actually load *all* items from the ZODB
>
>
> - experimental.catalogqueryplan (you will need to pin to the last Plone 3.x
> version which IIRC is 3.0.2
>
> - experimental.contentcreation
> This speeds up the act of initially creating new content (eg clicking
> 'page' from the add new menu). It puts in a 'dummy' catalog in portal_factory which
> prevents the object being indexed/unindexed as it is created
>
> - experimental.daterangeindexoptimisations
> You will need to re-index your catalog after this to take effect, but it optimises the
> date indexes for you.
>
> 2) Run the catalog optimise script:
> https://github.com/hannosch/scripts/commits/master/catalogoptimize.py
> This will optimise the BTree buckets in your catalog
>
> 3) Install collective.stats
> This will tell you if you 'reads' are actually doing 'writes' to your database and how many
> objects loaded and stored for each request.
>
> -Matt
>
> --
> Matt Hamilton [hidden email]
> Netsight Internet Solutions, Ltd. Business Vision on the Internet
> http://www.netsight.co.uk +44 (0)117 9090901
> Web Design | Zope/Plone Development and Consulting | Co-location | Hosting
>
>
>
>
>
> ------------------------------------------------------------------------------
> Learn Windows Azure Live! Tuesday, Dec 13, 2011
> Microsoft is holding a special Learn Windows Azure training event for
> developers. It will provide a great way to learn Windows Azure and what it
> provides. You can attend the event by watching it streamed LIVE online.
> Learn more at http://p.sf.net/sfu/ms-windowsazure
> _______________________________________________
> Plone-Users mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/plone-users