Quantcast

Sprint task? Fixing the Python situation on Unix platforms

classic Classic list List threaded Threaded
7 messages Options
Alexander Limi Alexander Limi
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Sprint task? Fixing the Python situation on Unix platforms

Sending this on behalf of Martin, since he has to leave for his flight back to the UK:

All,

Problem #1 for new users on Unix platforms: Using a non-isolated system
Python.

We're now in a position where most Linux and Mac users will have a binary
called python2.6 in their path, so we can have instructions like::

   $ python2.6 bootstrap.py
   $ bin/buildout

Yay. Except if you do that with a Python you didn't compile yourself, it
explodes in weird and wonderful ways. Compiling/installing your own Python
is a significant barrier (I witnessed this for about 1/3rd of the people in
my training course). And of course, this happens every time Limi tries to do anything Plone-related.

A good-enough solution for most people is to use virtualenv. So now, our
instructions become::

   $ sudo easy_install -U virtualenv
   $ virtualenv -p /usr/bin/python2.6 --no-site-packages ~/python
   $ ~/python/bin/python bootstrap.py
   $ bin/buildout

So, here's the rub: We can do this for people in the ``bootstrap.py``. Right
now, people just download the one from http://python-distribute.org, or copy
one from another buildout. So, what if we had::

   $ python2.6 plone-bootstrap.py
   $ bin/buildout

This would:

 * Sniff out a half-decent Python
 * Install virtualenv (ideally without needing sudo access)
 * Create a virtualenv without site packages, e.g. in a ``.python`` directory
 * Use its isolated python to do what bootstrap does now, so ``bin/buildout``
   uses this

Some key constraints/requirements

 * It has to be a single file - no package you install with setuptools
 * It has to be cross-platform (including Windows, Linux, Mac)
 * It should ideally not require a buildout.cfg in the current directory, as
   bootstrap.py does
 * It needs to print meaningful error messages if it can't figure out what
   to do in the environment
 * It may need some kind of knowledge of Plone versions to understand which
   Python versions are compatible

Regards,
Martin

--
Alex Limi · +limi · @limi · limi.net




------------------------------------------------------------------------------
RSA(R) Conference 2012
Save $700 by Nov 18
Register now
http://p.sf.net/sfu/rsa-sfdev2dev1
_______________________________________________
Plone-developers mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/plone-developers
Alexander Limi · http://limi.net

William Deegan William Deegan
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Sprint task? Fixing the Python situation on Unix platforms

Hi all,

Here's a real rough shot at this request.


basically you can use virtualenv to create a bootstrap.py script and add some of your own logic to it.
In the add on logic, I've had it install zc.buildout.

Should be trivial to pull a good boilerplate buildout.cfg from the web if none is present.

One gotcha right now, is I specify the created virtualenv dir to be .plone_python and so the buildout get's put in there as well.

If I specify the virtualenv dir as the current working dir, then virtualenv adds : etc lib bin include directories to the current directory.

Is that an issue?

Still many todo's to fulfill the whole request, but want to make sure such a solution isn't totally wrong.

-Bill


------------------------------------------------------------------------------
RSA(R) Conference 2012
Save $700 by Nov 18
Register now
http://p.sf.net/sfu/rsa-sfdev2dev1
_______________________________________________
Plone-developers mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/plone-developers
William Deegan William Deegan
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Sprint task? Fixing the Python situation on Unix platforms

In reply to this post by Alexander Limi
Hi all,

Here's a real rough shot at this request.


basically you can use virtualenv to create a bootstrap.py script and add some of your own logic to it.
In the add on logic, I've had it install zc.buildout.

Should be trivial to pull a good boilerplate buildout.cfg from the web if none is present.

One gotcha right now, is I specify the created virtualenv dir to be .plone_python and so the buildout get's put in there as well.

If I specify the virtualenv dir as the current working dir, then virtualenv adds : etc lib bin include directories to the current directory.

Is that an issue?

Still many todo's to fulfill the whole request, but want to make sure such a solution isn't totally wrong.

-Bill


------------------------------------------------------------------------------
RSA(R) Conference 2012
Save $700 by Nov 18
Register now
http://p.sf.net/sfu/rsa-sfdev2dev1
_______________________________________________
Plone-developers mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/plone-developers
Ricardo Newbery-2 Ricardo Newbery-2
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Sprint task? Fixing the Python situation on Unix platforms


On Nov 7, 2011, at 5:08 PM, William Deegan wrote:

> Hi all,
>
> Here's a real rough shot at this request.
>
> https://github.com/bdbaddog/Plone-bootstrap
>
> basically you can use virtualenv to create a bootstrap.py script and add some of your own logic to it.
> In the add on logic, I've had it install zc.buildout.
>
> Should be trivial to pull a good boilerplate buildout.cfg from the web if none is present.
>
> One gotcha right now, is I specify the created virtualenv dir to be .plone_python and so the buildout get's put in there as well.
>
> If I specify the virtualenv dir as the current working dir, then virtualenv adds : etc lib bin include directories to the current directory.
>
> Is that an issue?
>
> Still many todo's to fulfill the whole request, but want to make sure such a solution isn't totally wrong.
>
> -Bill



Thanks Bill.  I've got some comments I'll post in a following message but first I thought it might be useful to relay to the list some of the discussion that came up on Sunday before I had to catch my flight home.

There was some initial confusion about what audience we were targeting and where to put this virtualenv-bootstrap script which meant I probably started a lot of unnecessary angst about this proposal.

Regarding the target audience..

When I first read it, the initial proposal suggested the target audience was pretty much all "new users ... using a non-isolated system python".  Steve pointed out that all the official plone installers already take care of this issue.  So the actual target audience appears to be all new users who are not using one of the official plone installers.  This can include:

  1) Attendees at some Plone workshops.

  2) Developers working on add-ons or Plone core
     (an informal survey suggests that many of us don't
     use the installers for dev work).

  3) New users following one of the many instructions
     available, printed and online, on how to set up a
     Plone buildout without the installer.

  4) Any other user with a customized buildout
     configuration not using one of the installer
     frameworks.

Any of these audiences can be bitten by the python site-packages issue.

A side note: Another potential issue when using a system python are missing python dev libraries, but this can probably be lumped together with the general class of problems of missing system dependancies which this proposal doesn't address (although perhaps we can consider adding some system dependency checking for the common cases).

Another side note: The usecase I discussed in the lightning talk (http://www.slideshare.net/newbery/stupid-buildout-tricks) would be a variant of the #4 case above.


Regarding where to put this virtualenv-bootstrap...

Eventually, it sounds like we're talking about possibly including this script with all raw Plone buildouts but not the installers which already solve this issue.  I had a brief discussion with Cris Ewing about including it with the ZopeSkel Plone templates but that's a discussion we probably should delay until we've played with this a bit more.

There was some discussion about replacing the existing bootstrap.py with this script but concerns were raised about fundamentally changing what bootstrap.py does and the likelihood that many installations actually do require access to the system python site-packages.  For now, I think we're just talking about placing a alternative bootstrap alongside the existing bootstrap.py.


Final comment...

One alternative to virtualenv raised but we didn't have time to explore was the possibility of just using an updated zc.buildout version which supports site-packages isolation.  I've heard from several people that claim to have used version 1.5.x successfully in modern Plone buildouts.  There were some early issues when the 1.5 series came out and there might still be a far number of recipes out-in-the-wild which will break in 1.5 (see http://pypi.python.org/pypi/zc.buildout/1.5.2#recipes-that-support-a-system-python) so this might be a riskier alternative.


Cheers,
Ric



------------------------------------------------------------------------------
RSA(R) Conference 2012
Save $700 by Nov 18
Register now
http://p.sf.net/sfu/rsa-sfdev2dev1
_______________________________________________
Plone-developers mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/plone-developers
Ricardo Newbery-2 Ricardo Newbery-2
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Sprint task? Fixing the Python situation on Unix platforms

In reply to this post by William Deegan

On Nov 7, 2011, at 5:08 PM, William Deegan wrote:

> Hi all,
>
> Here's a real rough shot at this request.
>
> https://github.com/bdbaddog/Plone-bootstrap
>
> basically you can use virtualenv to create a bootstrap.py script and add some of your own logic to it.
> In the add on logic, I've had it install zc.buildout.


To clarify for the peanut gallery... This is just explaining how the script is initially created.  I believe the final product seen by the end user will be just a single plone-bootstrap.py file.  I'm assuming we'll just keep the generator script around to regenerate the bootstrap script if the virtualenv package is updated.



> Should be trivial to pull a good boilerplate buildout.cfg from the web if none is present.


I'm not sure what you mean here.  We don't care about the buildout.cfg.  That's going to be different in each case.



> One gotcha right now, is I specify the created virtualenv dir to be .plone_python and so the buildout get's put in there as well.


That should not be happening.  The location of the python does not determine the location of the buildout root.  That is defined by the location of the buildout config file (although it can be overridden on the command line).



> If I specify the virtualenv dir as the current working dir, then virtualenv adds : etc lib bin include directories to the current directory.
>
> Is that an issue?


I personally don't care.  In my version, I just made the working dir into the virtualenv.  But Martin's proposal was to place it into a subdir called ".python" which seems fine to me.




> Still many todo's to fulfill the whole request, but want to make sure such a solution isn't totally wrong.


It's looking great.  Just a couple of minor quibbles:

-1 on installing PIL during bootstrap.  PIL or Pillow should just be one of the eggs defined in the buildout config.  It seems out-of-scope for this effort.

The buildout script location is currently hardcoded to the 'bin' directory.  You may want to add a test for Windows as I believe the directory is "Script" in the Windows buildout.

Other than adding the python-sniffer, what else is left on the todo list?

Thanks for progress so far!

Cheers,
Ric



------------------------------------------------------------------------------
RSA(R) Conference 2012
Save $700 by Nov 18
Register now
http://p.sf.net/sfu/rsa-sfdev2dev1
_______________________________________________
Plone-developers mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/plone-developers
Dylan Jay Dylan Jay
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Sprint task? Fixing the Python situation on Unix platforms

In reply to this post by Alexander Limi
Not sure it helps at all but there is a bunch of code that does this  
same thing via fabric in collective.hostout.

https://github.com/collective/collective.hostout/blob/master/collective/hostout/fabfile.py 
#L409

In theory it should be possible to turn all remote fabric commands  
into local commands and then reuse this same code to create an  
isolated bootstrapped environment locally. I haven't tried to do this.  
Also it doesn't solve the problem where to get the fabfile in the  
first place. but the code is still there if it helps anyone :)

---
Dylan Jay
Technical Solutions Manager
PretaWeb: Multisite Performance Support
P: +612 80819071 | M: +61421477460 | twitter.com/djay75 | linkedin.com/
in/djay75

On 07/11/2011, at 9:07 AM, Alex Limi wrote:

> Sending this on behalf of Martin, since he has to leave for his  
> flight back to the UK:
>
> All,
>
> Problem #1 for new users on Unix platforms: Using a non-isolated  
> system
> Python.
>
> We're now in a position where most Linux and Mac users will have a  
> binary
> called python2.6 in their path, so we can have instructions like::
>
>    $ python2.6 bootstrap.py
>    $ bin/buildout
>
> Yay. Except if you do that with a Python you didn't compile  
> yourself, it
> explodes in weird and wonderful ways. Compiling/installing your own  
> Python
> is a significant barrier (I witnessed this for about 1/3rd of the  
> people in
> my training course). And of course, this happens every time Limi  
> tries to do anything Plone-related.
>
> A good-enough solution for most people is to use virtualenv. So now,  
> our
> instructions become::
>
>    $ sudo easy_install -U virtualenv
>    $ virtualenv -p /usr/bin/python2.6 --no-site-packages ~/python
>    $ ~/python/bin/python bootstrap.py
>    $ bin/buildout
>
> So, here's the rub: We can do this for people in the  
> ``bootstrap.py``. Right
> now, people just download the one from http://python-distribute.org,  
> or copy
> one from another buildout. So, what if we had::
>
>    $ python2.6 plone-bootstrap.py
>    $ bin/buildout
>
> This would:
>
>  * Sniff out a half-decent Python
>  * Install virtualenv (ideally without needing sudo access)
>  * Create a virtualenv without site packages, e.g. in a ``.python``  
> directory
>  * Use its isolated python to do what bootstrap does now, so ``bin/
> buildout``
>    uses this
>
> Some key constraints/requirements
>
>  * It has to be a single file - no package you install with setuptools
>  * It has to be cross-platform (including Windows, Linux, Mac)
>  * It should ideally not require a buildout.cfg in the current  
> directory, as
>    bootstrap.py does
>  * It needs to print meaningful error messages if it can't figure  
> out what
>    to do in the environment
>  * It may need some kind of knowledge of Plone versions to  
> understand which
>    Python versions are compatible
>
> Regards,
> Martin
>
> --
> Alex Limi · +limi · @limi · limi.net
>
>
>
> ------------------------------------------------------------------------------
> RSA(R) Conference 2012
> Save $700 by Nov 18
> Register now
> http://p.sf.net/sfu/rsa-sfdev2dev1_______________________________________________
> Plone-developers mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/plone-developers


------------------------------------------------------------------------------
RSA(R) Conference 2012
Save $700 by Nov 18
Register now
http://p.sf.net/sfu/rsa-sfdev2dev1
_______________________________________________
Plone-developers mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/plone-developers
Martin Aspeli Martin Aspeli
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Sprint task? Fixing the Python situation on Unix platforms

In reply to this post by Ricardo Newbery-2
Hi,

On 8 November 2011 03:59, Ricardo Newbery <[hidden email]> wrote:

> Regarding the target audience..
>
> When I first read it, the initial proposal suggested the target audience was pretty much all "new users ... using a non-isolated system python".  Steve pointed out that all the official plone installers already take care of this issue.  So the actual target audience appears to be all new users who are not using one of the official plone installers.  This can include:
>
>  1) Attendees at some Plone workshops.
>
>  2) Developers working on add-ons or Plone core
>     (an informal survey suggests that many of us don't
>     use the installers for dev work).
>
>  3) New users following one of the many instructions
>     available, printed and online, on how to set up a
>     Plone buildout without the installer.
>
>  4) Any other user with a customized buildout
>     configuration not using one of the installer
>     frameworks.
>
> Any of these audiences can be bitten by the python site-packages issue.

Right. I think we've had an assumption that people 'graduating' from
installers to custom buildout are capable of isolating their own
system Python, but I think that's not really the case. It's easy to
forget, if nothing else.

Maybe we should push people towards using the installers for more
things, but that may be at bets a partial solution. Hence, I think if
we can make the bootstrap step of the standard buildout process - or
buildout itself - be more intelligent about global site packages,
there's not really a downside in doing so.

> A side note: Another potential issue when using a system python are missing python dev libraries, but this can probably be lumped together with the general class of problems of missing system dependancies which this proposal doesn't address (although perhaps we can consider adding some system dependency checking for the common cases).

Good point, though probably not solvable with this, so a separate discussion.

> Regarding where to put this virtualenv-bootstrap...
>
> Eventually, it sounds like we're talking about possibly including this script with all raw Plone buildouts but not the installers which already solve this issue.  I had a brief discussion with Cris Ewing about including it with the ZopeSkel Plone templates but that's a discussion we probably should delay until we've played with this a bit more.

Let's make it work first, but yeah, that'd be the idea.

> There was some discussion about replacing the existing bootstrap.py with this script but concerns were raised about fundamentally changing what bootstrap.py does and the likelihood that many installations actually do require access to the system python site-packages.  For now, I think we're just talking about placing a alternative bootstrap alongside the existing bootstrap.py.

+1

> Final comment...
>
> One alternative to virtualenv raised but we didn't have time to explore was the possibility of just using an updated zc.buildout version which supports site-packages isolation.  I've heard from several people that claim to have used version 1.5.x successfully in modern Plone buildouts.  There were some early issues when the 1.5 series came out and there might still be a far number of recipes out-in-the-wild which will break in 1.5 (see http://pypi.python.org/pypi/zc.buildout/1.5.2#recipes-that-support-a-system-python) so this might be a riskier alternative.

I saw this too, after I wrote the mail that Limi sent on my behalf. I
think we do need to get to 1.5 eventually, so maybe testing that is a
good use of time in the short term. If it solves the problem, then
that's in many ways simpler.

Beyond that, I think a virtualenv-bootstrap.py script needs to be very
foolproof. Only one step ("python virtual-bootstrap.py"), no
configuration options, and producing a very standard buildout layout
so as not to invalidate old documentation.

Martin

------------------------------------------------------------------------------
RSA(R) Conference 2012
Save $700 by Nov 18
Register now
http://p.sf.net/sfu/rsa-sfdev2dev1
_______________________________________________
Plone-developers mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/plone-developers
Loading...