Quantcast

Re: [Plone-developers] jenkins and git repository monitoring

classic Classic list List threaded Threaded
4 messages Options
Timo Stollenwerk Timo Stollenwerk
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [Plone-developers] jenkins and git repository monitoring

Hi Patrick,

I'm not sure if polling github is really what we want for CI (the
ScriptTrigger plugin does poll after all if I understand it correctly).
Polling works fine for smaller projects, but I don't think it does
really scale for a project like Plone. With a large number of jobs
polling this is a waste of network resources and we don't really get
atomic commits (which is not really CI then).

I gave this some thought and maybe we can set up some kind of
"dispatcher"-Job in Jenkins that is triggered for each commit on a
coredev package (it seems like there is no way to trigger a remote URL
for the entire Plone project on github, so we have to set up a trigger
for each coredev package).

The "dispatcher" could be a regular Jenkins job that we set up for each
coredev branch (4.0, 4.1, 4.2), that only succeeds if the commit is
relevant for that particular coredev branch (the Jenkins job could
include a bash script or buildout recipe that reads the eggs under
development to make this decision). If a "dispatcher"-job is successful,
it triggers the real build, tests, coverage, etc..

I think a proof-of-concept would be easy to setup. Another option would
be to implement the dispatcher as a small app, maybe with Pyramid on GAE
(but this app would have to read the coredev buildouts to decide which
package is relevant for a particular buildout).

We set up a github project at the Plone conference to work on the
Jenkins setup and this particular problem:

https://github.com/plone/jenkins.plone.org

Feel free to join our efforts. :)

Cheers,
timo

Am 20.11.2011 02:25, schrieb Patrick Gerken:

> Hi,
>
> Jenkins was stuck for a week and by cancelling some job I got it
> running again. But now it runs new tests for Plone 4.1 and 4.2 every
> 20 minutes.
> That's because the git plugin for jenkins really can't handle polling
> for changes for multiple repositories for a single project.
> An alternative seems to be to use this plugin:
>
> https://wiki.jenkins-ci.org/display/JENKINS/ScriptTrigger+Plugin
>
> and set
>
> ./bin/develop up -v |  grep -e '^Updat\(ing\|ed\)'
>
> as the script
>
> The plugin is currently missing and I can't add it, but I'd be glad to help
>
> Best regards,
>
>    Patrick
>
> ------------------------------------------------------------------------------
> All the data continuously generated in your IT infrastructure
> contains a definitive record of customers, application performance,
> security threats, fraudulent activity, and more. Splunk takes this
> data and makes sense of it. IT sense. And common sense.
> http://p.sf.net/sfu/splunk-novd2d
> _______________________________________________
> Plone-developers mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/plone-developers


------------------------------------------------------------------------------
All the data continuously generated in your IT infrastructure
contains a definitive record of customers, application performance,
security threats, fraudulent activity, and more. Splunk takes this
data and makes sense of it. IT sense. And common sense.
http://p.sf.net/sfu/splunk-novd2d
_______________________________________________
Plone-website mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/plone-website
The Wiki: http://plone.org/development/current/projects/WebsiteTeam
William Deegan William Deegan
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [Plone-developers] jenkins and git repository monitoring

All,
On Nov 20, 2011, at 7:59 AM, Patrick Gerken wrote:

> On Sun, Nov 20, 2011 at 15:51, Timo Stollenwerk <[hidden email]> wrote:
>> Hi Patrick,
>>
>> I'm not sure if polling github is really what we want for CI (the
>> ScriptTrigger plugin does poll after all if I understand it correctly).
>> Polling works fine for smaller projects, but I don't think it does
>> really scale for a project like Plone. With a large number of jobs
>> polling this is a waste of network resources and we don't really get
>> atomic commits (which is not really CI then).
>
> Yes, it's really slow, right now it takes 20 minutes. We are polling
> github and svn right now, and the git plugin is too stupid to work
> with multiple checkouts reliably, and believes every time something
> has changed.
>
> My idea would be a small improvement of the current situation, while
> yours below certainly is better, but it needs to be done.
> I'd love to join the efforts, but right now I can't even keep up with
> mail traffic :-(

Was wondering what's the current status on this.
Apparently github has a API which would allow us to walk all the plone repos and set the hook programmatically
http://developer.github.com/v3/repos/hooks/

Would that resolve some of these issues?
Likewise would a commit hook on the plone SVN repo be possible?
So no more polling, just responding to posts?

-Bill
------------------------------------------------------------------------------
Try before you buy = See our experts in action!
The most comprehensive online learning library for Microsoft developers
is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, CSS3, MVC3,
Metro Style Apps, more. Free future releases when you subscribe now!
http://p.sf.net/sfu/learndevnow-dev2
_______________________________________________
Plone-website mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/plone-website
The Wiki: http://plone.org/development/current/projects/WebsiteTeam
Timo Stollenwerk Timo Stollenwerk
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [Plone-developers] jenkins and git repository monitoring

Am 03.02.2012 23:07, schrieb William Deegan:

> All,
> On Nov 20, 2011, at 7:59 AM, Patrick Gerken wrote:
>
>> On Sun, Nov 20, 2011 at 15:51, Timo Stollenwerk <[hidden email]> wrote:
>>> Hi Patrick,
>>>
>>> I'm not sure if polling github is really what we want for CI (the
>>> ScriptTrigger plugin does poll after all if I understand it correctly).
>>> Polling works fine for smaller projects, but I don't think it does
>>> really scale for a project like Plone. With a large number of jobs
>>> polling this is a waste of network resources and we don't really get
>>> atomic commits (which is not really CI then).
>>
>> Yes, it's really slow, right now it takes 20 minutes. We are polling
>> github and svn right now, and the git plugin is too stupid to work
>> with multiple checkouts reliably, and believes every time something
>> has changed.
>>
>> My idea would be a small improvement of the current situation, while
>> yours below certainly is better, but it needs to be done.
>> I'd love to join the efforts, but right now I can't even keep up with
>> mail traffic :-(
>
> Was wondering what's the current status on this.
> Apparently github has a API which would allow us to walk all the plone repos and set the hook programmatically
> http://developer.github.com/v3/repos/hooks/

I added a github user to jenkins.plone.org so we can trigger builds on
github commits. I tested it for several core projects and it works
pretty well (for single packages).

As David already pointed out, we are also able to set github post-commit
hooks programmatically.

The question is still, how do we decide when to trigger a coredev build
and how do we set up the Jenkins jobs. After discussing this issue with
Eric I wrote a proposal for a Jenkins job architecture:

https://github.com/plone/jenkins.plone.org/blob/master/architecture.rst

This is still work in progress, so I'm open for discussion. Feel free to
add your ideas and thoughts to that document.

I think the only way we can figure out a good way to set up the coredev
jobs is to try out different approaches. Therefore I started to play
around a bit the Jenkins API to set up jobs automatically (because we
don't want to set up jobs for all core packages manually):

https://github.com/plone/jenkins.plone.org/blob/master/fabfile.py

Michael Mulich also worked on setting up Jenkins jobs automatically.

@Michael: Could you give us a quick update on what you did so far?

I'm constantly working on the Jenkins setup, but right now I can only
spare a couple of hours every week. I hope to get some work done at the
Plone Konferenz sprint in Munich later this month.

FYI: I also worked on buildout.jenkins this weekend and it is now
possible to run code coverage and code analysis on multiple packages:

https://github.com/plone/buildout.jenkins

Cheers,
timo




------------------------------------------------------------------------------
Try before you buy = See our experts in action!
The most comprehensive online learning library for Microsoft developers
is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, CSS3, MVC3,
Metro Style Apps, more. Free future releases when you subscribe now!
http://p.sf.net/sfu/learndevnow-dev2
_______________________________________________
Plone-website mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/plone-website
The Wiki: http://plone.org/development/current/projects/WebsiteTeam
Timo Stollenwerk Timo Stollenwerk
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [Plone-developers] jenkins and git repository monitoring

Hi Laurence,

Am 06.02.2012 15:02, schrieb Laurence Rowe:
> We really should be running all tests across all packages following a
> push to any package. For example, it's possible that a change to
> plone.app.layout might cause a failure in a Products.CMFPlone test and
> we want to catch that as early as possible. The key advantage of
> Continuous Integration is that these failures are caught, everyone
> should already be running the tests on the package they're making
> changes too locally before committing. CI lets us catch those knock-on
> failures that might not otherwise be noticed.

I agree. Running all tests across all packages on any commit would be
desirable for exactly the reasons you pointed out. The more I think
about it, the more important it seems to me that we really run all tests
on each commit. CI only works if regressions are catched right away.

The reason why we thought about splitting up the build and running tests
on the packages itself (and run the full tests on an hourly basis), was
that we ran into performance issues with our two CI servers.

To solve this, we can either throw more hardware at our CI setup or find
a smart way to decide on which commits we have to trigger which jobs, to
reduce the number of jobs that Jenkins has to run. Right now we have to
run all three Plone coredev jobs with all tests for every commit on any
github/svn package that is in one of these buildouts. This seems to be
too much for our current setup and we want to run code analysis and
other coredev and add-on product jobs in the future as well.

Cheers,
timo

------------------------------------------------------------------------------
Virtualization & Cloud Management Using Capacity Planning
Cloud computing makes use of virtualization - but cloud computing
also focuses on allowing computing to be delivered as a service.
http://www.accelacomm.com/jaw/sfnl/114/51521223/
_______________________________________________
Plone-website mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/plone-website
The Wiki: http://plone.org/development/current/projects/WebsiteTeam
Loading...