|
Timo Stollenwerk |
|
|
Hi Patrick,
I'm not sure if polling github is really what we want for CI (the ScriptTrigger plugin does poll after all if I understand it correctly). Polling works fine for smaller projects, but I don't think it does really scale for a project like Plone. With a large number of jobs polling this is a waste of network resources and we don't really get atomic commits (which is not really CI then). I gave this some thought and maybe we can set up some kind of "dispatcher"-Job in Jenkins that is triggered for each commit on a coredev package (it seems like there is no way to trigger a remote URL for the entire Plone project on github, so we have to set up a trigger for each coredev package). The "dispatcher" could be a regular Jenkins job that we set up for each coredev branch (4.0, 4.1, 4.2), that only succeeds if the commit is relevant for that particular coredev branch (the Jenkins job could include a bash script or buildout recipe that reads the eggs under development to make this decision). If a "dispatcher"-job is successful, it triggers the real build, tests, coverage, etc.. I think a proof-of-concept would be easy to setup. Another option would be to implement the dispatcher as a small app, maybe with Pyramid on GAE (but this app would have to read the coredev buildouts to decide which package is relevant for a particular buildout). We set up a github project at the Plone conference to work on the Jenkins setup and this particular problem: https://github.com/plone/jenkins.plone.org Feel free to join our efforts. :) Cheers, timo Am 20.11.2011 02:25, schrieb Patrick Gerken: > Hi, > > Jenkins was stuck for a week and by cancelling some job I got it > running again. But now it runs new tests for Plone 4.1 and 4.2 every > 20 minutes. > That's because the git plugin for jenkins really can't handle polling > for changes for multiple repositories for a single project. > An alternative seems to be to use this plugin: > > https://wiki.jenkins-ci.org/display/JENKINS/ScriptTrigger+Plugin > > and set > > ./bin/develop up -v | grep -e '^Updat\(ing\|ed\)' > > as the script > > The plugin is currently missing and I can't add it, but I'd be glad to help > > Best regards, > > Patrick > > ------------------------------------------------------------------------------ > All the data continuously generated in your IT infrastructure > contains a definitive record of customers, application performance, > security threats, fraudulent activity, and more. Splunk takes this > data and makes sense of it. IT sense. And common sense. > http://p.sf.net/sfu/splunk-novd2d > _______________________________________________ > Plone-developers mailing list > [hidden email] > https://lists.sourceforge.net/lists/listinfo/plone-developers ------------------------------------------------------------------------------ All the data continuously generated in your IT infrastructure contains a definitive record of customers, application performance, security threats, fraudulent activity, and more. Splunk takes this data and makes sense of it. IT sense. And common sense. http://p.sf.net/sfu/splunk-novd2d _______________________________________________ Plone-website mailing list [hidden email] https://lists.sourceforge.net/lists/listinfo/plone-website The Wiki: http://plone.org/development/current/projects/WebsiteTeam |
|
William Deegan |
|
|
All,
On Nov 20, 2011, at 7:59 AM, Patrick Gerken wrote: > On Sun, Nov 20, 2011 at 15:51, Timo Stollenwerk <[hidden email]> wrote: >> Hi Patrick, >> >> I'm not sure if polling github is really what we want for CI (the >> ScriptTrigger plugin does poll after all if I understand it correctly). >> Polling works fine for smaller projects, but I don't think it does >> really scale for a project like Plone. With a large number of jobs >> polling this is a waste of network resources and we don't really get >> atomic commits (which is not really CI then). > > Yes, it's really slow, right now it takes 20 minutes. We are polling > github and svn right now, and the git plugin is too stupid to work > with multiple checkouts reliably, and believes every time something > has changed. > > My idea would be a small improvement of the current situation, while > yours below certainly is better, but it needs to be done. > I'd love to join the efforts, but right now I can't even keep up with > mail traffic :-( Was wondering what's the current status on this. Apparently github has a API which would allow us to walk all the plone repos and set the hook programmatically http://developer.github.com/v3/repos/hooks/ Would that resolve some of these issues? Likewise would a commit hook on the plone SVN repo be possible? So no more polling, just responding to posts? -Bill ------------------------------------------------------------------------------ Try before you buy = See our experts in action! The most comprehensive online learning library for Microsoft developers is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, CSS3, MVC3, Metro Style Apps, more. Free future releases when you subscribe now! http://p.sf.net/sfu/learndevnow-dev2 _______________________________________________ Plone-website mailing list [hidden email] https://lists.sourceforge.net/lists/listinfo/plone-website The Wiki: http://plone.org/development/current/projects/WebsiteTeam |
|
Timo Stollenwerk |
|
|
Am 03.02.2012 23:07, schrieb William Deegan:
> All, > On Nov 20, 2011, at 7:59 AM, Patrick Gerken wrote: > >> On Sun, Nov 20, 2011 at 15:51, Timo Stollenwerk <[hidden email]> wrote: >>> Hi Patrick, >>> >>> I'm not sure if polling github is really what we want for CI (the >>> ScriptTrigger plugin does poll after all if I understand it correctly). >>> Polling works fine for smaller projects, but I don't think it does >>> really scale for a project like Plone. With a large number of jobs >>> polling this is a waste of network resources and we don't really get >>> atomic commits (which is not really CI then). >> >> Yes, it's really slow, right now it takes 20 minutes. We are polling >> github and svn right now, and the git plugin is too stupid to work >> with multiple checkouts reliably, and believes every time something >> has changed. >> >> My idea would be a small improvement of the current situation, while >> yours below certainly is better, but it needs to be done. >> I'd love to join the efforts, but right now I can't even keep up with >> mail traffic :-( > > Was wondering what's the current status on this. > Apparently github has a API which would allow us to walk all the plone repos and set the hook programmatically > http://developer.github.com/v3/repos/hooks/ I added a github user to jenkins.plone.org so we can trigger builds on github commits. I tested it for several core projects and it works pretty well (for single packages). As David already pointed out, we are also able to set github post-commit hooks programmatically. The question is still, how do we decide when to trigger a coredev build and how do we set up the Jenkins jobs. After discussing this issue with Eric I wrote a proposal for a Jenkins job architecture: https://github.com/plone/jenkins.plone.org/blob/master/architecture.rst This is still work in progress, so I'm open for discussion. Feel free to add your ideas and thoughts to that document. I think the only way we can figure out a good way to set up the coredev jobs is to try out different approaches. Therefore I started to play around a bit the Jenkins API to set up jobs automatically (because we don't want to set up jobs for all core packages manually): https://github.com/plone/jenkins.plone.org/blob/master/fabfile.py Michael Mulich also worked on setting up Jenkins jobs automatically. @Michael: Could you give us a quick update on what you did so far? I'm constantly working on the Jenkins setup, but right now I can only spare a couple of hours every week. I hope to get some work done at the Plone Konferenz sprint in Munich later this month. FYI: I also worked on buildout.jenkins this weekend and it is now possible to run code coverage and code analysis on multiple packages: https://github.com/plone/buildout.jenkins Cheers, timo ------------------------------------------------------------------------------ Try before you buy = See our experts in action! The most comprehensive online learning library for Microsoft developers is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, CSS3, MVC3, Metro Style Apps, more. Free future releases when you subscribe now! http://p.sf.net/sfu/learndevnow-dev2 _______________________________________________ Plone-website mailing list [hidden email] https://lists.sourceforge.net/lists/listinfo/plone-website The Wiki: http://plone.org/development/current/projects/WebsiteTeam |
|
Timo Stollenwerk |
|
|
Hi Laurence,
Am 06.02.2012 15:02, schrieb Laurence Rowe: > We really should be running all tests across all packages following a > push to any package. For example, it's possible that a change to > plone.app.layout might cause a failure in a Products.CMFPlone test and > we want to catch that as early as possible. The key advantage of > Continuous Integration is that these failures are caught, everyone > should already be running the tests on the package they're making > changes too locally before committing. CI lets us catch those knock-on > failures that might not otherwise be noticed. I agree. Running all tests across all packages on any commit would be desirable for exactly the reasons you pointed out. The more I think about it, the more important it seems to me that we really run all tests on each commit. CI only works if regressions are catched right away. The reason why we thought about splitting up the build and running tests on the packages itself (and run the full tests on an hourly basis), was that we ran into performance issues with our two CI servers. To solve this, we can either throw more hardware at our CI setup or find a smart way to decide on which commits we have to trigger which jobs, to reduce the number of jobs that Jenkins has to run. Right now we have to run all three Plone coredev jobs with all tests for every commit on any github/svn package that is in one of these buildouts. This seems to be too much for our current setup and we want to run code analysis and other coredev and add-on product jobs in the future as well. Cheers, timo ------------------------------------------------------------------------------ Virtualization & Cloud Management Using Capacity Planning Cloud computing makes use of virtualization - but cloud computing also focuses on allowing computing to be delivered as a service. http://www.accelacomm.com/jaw/sfnl/114/51521223/ _______________________________________________ Plone-website mailing list [hidden email] https://lists.sourceforge.net/lists/listinfo/plone-website The Wiki: http://plone.org/development/current/projects/WebsiteTeam |
| Powered by Nabble | Edit this page |