plone.app.toolbar UI testing using Robot Framework and Selenium

classic Classic list List threaded Threaded
26 messages Options
12
Ed Manlove-2 Ed Manlove-2
Reply | Threaded
Open this post in threaded view
|

plone.app.toolbar UI testing using Robot Framework and Selenium

In preparation for the upcoming Sea Sprint, I wanted to help by
providing some examples and documentation of using
robotframework/selenium with Plone as a testing framework.  In terms of
examples/code, I want to provide a buildout configuration along with
some common page objects for the toolbar so effort can be made to focus
on testing and not testing framework. For documentation I wanted to make
it easy for developers to get started and provide any hints and help to
workaround testing framework issues so that again focus can be on
testing and building plone.app.toolbar.

Below as well as on my github site, https://github.com/emanlove, you
will find some initial work.  I will be posting these notes also to the
coactivate page shortly.

At this point I've forked buildout.deco and created a buildout
configuration file, pybot-act.cfg. I've also forked plone.app.toolbar to
add a test directory and sample tests (only two very basic ones exist as
of yet). Here is the steps to reproduce...

~$ mkdir s2l-plone
~$ cd s2l-plone
~/s2l-plone$ virtualenv -p /usr/bin/python2.6 --no-site-packages
clean-python26-envRunning virtualenv with interpreter /usr/bin/python2.6
~/s2l-plone $ source clean-python26-env/bin/activate
(clean-python26-env) ~/s2l-plone $ easy_install PIL
(clean-python26-env) ~/s2l-plone $ git clone
[hidden email]:emanlove/buildout.deco.git deco
(clean-python26-env) ~/s2l-plone $ cd deco/
(clean-python26-env) ~/s2l-plone/deco $ python bootstrap.py
(clean-python26-env) ~/s2l-plone/deco $ cp buildout.cfg.in buildout.cfg

and uncomment out the line 'toolbar-2.0.cfg' within buildout.cfg

(clean-python26-env) ~/s2l-plone/deco $ ./bin/buildout -c pybot-act.cfg

To run the two sample tests within my fork of plone.app.toolbar (which
the above buildout checks out instead of plone/plone.app.toolbar) use
the following command

(clean-python26-env) ~/s2l-plone/deco $ ./bin/pybot --variable
PORT:55001 src/plone.app.toolbar/acceptance-tests

Note there is an bug/error within the breakdown of zodb/zserver that I
will be looking at so as yet the this test file fails on teardown.  This
effects some error reporting and possible issue with having more than
one test file.  So just be aware it is there, I know about and am going
to look into it.

Also when testing within the browser I find it very helpful to get the
browser up so I can click around and use tools like firebug. Debugging
through robotframework has it own problems but we do want a similar
browser as that created by our test environment. So to get a test
browser I use zopepy.  Here is an example

(clean-python26-env) ~/s2l-plone/deco $ ./bin/zopepy

 >>> from plone.act import Zope2ServerLibrary
 >>> z2sl = Zope2ServerLibrary()
 >>>
z2sl.start_zope_server('plone.app.toolbar.testing.TOOLBAR_ROBOTFRAMEWORK_TESTING')
...
 >>> z2sl.zodb_setup()
...
 >>> from Selenium2Library import Selenium2Library
 >>> s2l = Selenium2Library()
 >>> s2l.open_browser('localhost:55001/plone')
1

Now you should have the browser using the p.a.toolbar layer for which
you can explore/debug/test/etc.  To shut down I type

 >>> s2l.close_browser()
 >>> z2sl.stop_zope_server()
Traceback (most recent call last):
... # here is the foremention suite shutdown issue that I will be
looking into

I welcome any feedback with the above instructions and can go into
further detail of the various aspects of this work. Just ask.

Ed

------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and
threat landscape has changed and how IT managers can respond. Discussions
will include endpoint security, mobile security and the latest in malware
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
_______________________________________________
Plone-developers mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/plone-developers
Chris Calloway Chris Calloway
Reply | Threaded
Open this post in threaded view
|

Re: plone.app.toolbar UI testing using Robot Framework and Selenium

On 9/7/2012 11:26 AM, Ed Manlove wrote:
> In preparation for the upcoming Sea Sprint, I wanted to help by
> providing some examples and documentation of using
> robotframework/selenium with Plone as a testing framework.

This is great, Ed. Thanks.

--
Sincerely,

Chris Calloway http://nccoos.org/Members/cbc
office: 3313 Venable Hall   phone: (919) 599-3530
mail: Campus Box #3300, UNC-CH, Chapel Hill, NC 27599

------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and
threat landscape has changed and how IT managers can respond. Discussions
will include endpoint security, mobile security and the latest in malware
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
_______________________________________________
Plone-developers mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/plone-developers
Timo Stollenwerk Timo Stollenwerk
Reply | Threaded
Open this post in threaded view
|

Re: plone.app.toolbar UI testing using Robot Framework and Selenium

In reply to this post by Ed Manlove-2
Hi Ed,

thank you so much for working on this! Since the last plone conf I
haven't had the time to look into it.

Did anybody work on or look into the integration of robot framework into
plone.app.testing recently? I like robot framework and I think its
BDD-tests can really make functional testing in Plone a lot easier.
Though, the nicest robot framework tests won't really help us if we
can't run them in a stable and reliable manner. For this, we need the
integration into plone.app.testing. Otherwise we don't even have to
start trying to run those tests continuously on jenkins.plone.org.

I started to play around a bit with gocept.selenium for testing
plone.app.collection. It comes with a with a plone.app.testing layer, so
test setup/teardown is not an isse:

https://github.com/plone/plone.app.collection/blob/selenium/plone/app/collection/tests/test_functional.py

(I saw that plone.app.testing itself also has a, yet undocumented,
selenium layer, so this also might be worth a look.)

I'm currently trying to run those selenium tests on jenkins.plone.org, I
will keep you posted about my progress. Whatever framework we use we
will have to run the tests on jenkins.

The bottom line in my opinion is that we desperately need a way to test
our code with selenium (not only for deco) and we need it NOW. If we can
make this work with robot, great! If not we should choose whatever works
for us right now.

Cheers,
timo

Am 07.09.2012 17:26, schrieb Ed Manlove:

> In preparation for the upcoming Sea Sprint, I wanted to help by
> providing some examples and documentation of using
> robotframework/selenium with Plone as a testing framework.  In terms of
> examples/code, I want to provide a buildout configuration along with
> some common page objects for the toolbar so effort can be made to focus
> on testing and not testing framework. For documentation I wanted to make
> it easy for developers to get started and provide any hints and help to
> workaround testing framework issues so that again focus can be on
> testing and building plone.app.toolbar.
>
> Below as well as on my github site, https://github.com/emanlove, you
> will find some initial work.  I will be posting these notes also to the
> coactivate page shortly.
>
> At this point I've forked buildout.deco and created a buildout
> configuration file, pybot-act.cfg. I've also forked plone.app.toolbar to
> add a test directory and sample tests (only two very basic ones exist as
> of yet). Here is the steps to reproduce...
>
> ~$ mkdir s2l-plone
> ~$ cd s2l-plone
> ~/s2l-plone$ virtualenv -p /usr/bin/python2.6 --no-site-packages
> clean-python26-envRunning virtualenv with interpreter /usr/bin/python2.6
> ~/s2l-plone $ source clean-python26-env/bin/activate
> (clean-python26-env) ~/s2l-plone $ easy_install PIL
> (clean-python26-env) ~/s2l-plone $ git clone
> [hidden email]:emanlove/buildout.deco.git deco
> (clean-python26-env) ~/s2l-plone $ cd deco/
> (clean-python26-env) ~/s2l-plone/deco $ python bootstrap.py
> (clean-python26-env) ~/s2l-plone/deco $ cp buildout.cfg.in buildout.cfg
>
> and uncomment out the line 'toolbar-2.0.cfg' within buildout.cfg
>
> (clean-python26-env) ~/s2l-plone/deco $ ./bin/buildout -c pybot-act.cfg
>
> To run the two sample tests within my fork of plone.app.toolbar (which
> the above buildout checks out instead of plone/plone.app.toolbar) use
> the following command
>
> (clean-python26-env) ~/s2l-plone/deco $ ./bin/pybot --variable
> PORT:55001 src/plone.app.toolbar/acceptance-tests
>
> Note there is an bug/error within the breakdown of zodb/zserver that I
> will be looking at so as yet the this test file fails on teardown.  This
> effects some error reporting and possible issue with having more than
> one test file.  So just be aware it is there, I know about and am going
> to look into it.
>
> Also when testing within the browser I find it very helpful to get the
> browser up so I can click around and use tools like firebug. Debugging
> through robotframework has it own problems but we do want a similar
> browser as that created by our test environment. So to get a test
> browser I use zopepy.  Here is an example
>
> (clean-python26-env) ~/s2l-plone/deco $ ./bin/zopepy
>
>   >>> from plone.act import Zope2ServerLibrary
>   >>> z2sl = Zope2ServerLibrary()
>   >>>
> z2sl.start_zope_server('plone.app.toolbar.testing.TOOLBAR_ROBOTFRAMEWORK_TESTING')
> ...
>   >>> z2sl.zodb_setup()
> ...
>   >>> from Selenium2Library import Selenium2Library
>   >>> s2l = Selenium2Library()
>   >>> s2l.open_browser('localhost:55001/plone')
> 1
>
> Now you should have the browser using the p.a.toolbar layer for which
> you can explore/debug/test/etc.  To shut down I type
>
>   >>> s2l.close_browser()
>   >>> z2sl.stop_zope_server()
> Traceback (most recent call last):
> ... # here is the foremention suite shutdown issue that I will be
> looking into
>
> I welcome any feedback with the above instructions and can go into
> further detail of the various aspects of this work. Just ask.
>
> Ed
>
> ------------------------------------------------------------------------------
> Live Security Virtual Conference
> Exclusive live event will cover all the ways today's security and
> threat landscape has changed and how IT managers can respond. Discussions
> will include endpoint security, mobile security and the latest in malware
> threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
> _______________________________________________
> Plone-developers mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/plone-developers
>


------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and
threat landscape has changed and how IT managers can respond. Discussions
will include endpoint security, mobile security and the latest in malware
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
_______________________________________________
Plone-developers mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/plone-developers
Asko Soukka Asko Soukka
Reply | Threaded
Open this post in threaded view
|

Re: plone.app.toolbar UI testing using Robot Framework and Selenium

Timo Stollenwerk wrote:
> Did anybody work on or look into the integration of robot framework into
> plone.app.testing recently?

Me and Godefroid looked into it a few months ago and came up with
different solutions.

My solution was to wrap RF tests into Python unittest suites similarly
to how doctests have been wrapped (that became 'robotsuite'):

http://datakurre.pandala.org/2012/09/getting-started-with-robotframework-and.html

Godefroid wrote RF keywords ('Zope2ServerLibrary' in 'plone.act') to set
up and tear down p.a.testing layer directly in RF test suites. Ed just
described, how to experiment with those keywords, but you can also learn
how Godefroid used them in collective.jekyll:

https://github.com/gotcha/collective.jekyll/blob/master/src/collective/jekyll/testing.py
https://github.com/gotcha/collective.jekyll/blob/master/acceptance-tests/__init__.txt

Tests using 'Zope2ServerLibrary' can be run directly using pybot, which
has more features than zope.testrunner. Tests using 'robotsuite' can be
run with other 'zope.testrunner' tests.

-Asko

> Am 07.09.2012 17:26, schrieb Ed Manlove:
>> Also when testing within the browser I find it very helpful to get the
>> browser up so I can click around and use tools like firebug. Debugging
>> through robotframework has it own problems but we do want a similar
>> browser as that created by our test environment. So to get a test
>> browser I use zopepy.  Here is an example
>>
>> (clean-python26-env) ~/s2l-plone/deco $ ./bin/zopepy
>>
>>    >>>  from plone.act import Zope2ServerLibrary
>>    >>>  z2sl = Zope2ServerLibrary()
>>    >>>
>> z2sl.start_zope_server('plone.app.toolbar.testing.TOOLBAR_ROBOTFRAMEWORK_TESTING')
>> ...
>>    >>>  z2sl.zodb_setup()
>> ...
>>    >>>  from Selenium2Library import Selenium2Library
>>    >>>  s2l = Selenium2Library()
>>    >>>  s2l.open_browser('localhost:55001/plone')
>> 1
>>
>> Now you should have the browser using the p.a.toolbar layer for which
>> you can explore/debug/test/etc.  To shut down I type
>>
>>    >>>  s2l.close_browser()
>>    >>>  z2sl.stop_zope_server()

------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and
threat landscape has changed and how IT managers can respond. Discussions
will include endpoint security, mobile security and the latest in malware
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
_______________________________________________
Plone-developers mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/plone-developers
Timo Stollenwerk Timo Stollenwerk
Reply | Threaded
Open this post in threaded view
|

Re: plone.app.toolbar UI testing using Robot Framework and Selenium

Am 08.09.2012 10:20, schrieb Asko Soukka:
> Timo Stollenwerk wrote:
>> Did anybody work on or look into the integration of robot framework into
>> plone.app.testing recently?
>
> Me and Godefroid looked into it a few months ago and came up with
> different solutions.

Awesome! I didn't know we made so much progress lately. Can't wait to
look into it!

Do you already have experience with running those tests on Jenkins?

Cheers,
timo

------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and
threat landscape has changed and how IT managers can respond. Discussions
will include endpoint security, mobile security and the latest in malware
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
_______________________________________________
Plone-developers mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/plone-developers
Asko Soukka Asko Soukka
Reply | Threaded
Open this post in threaded view
|

Re: plone.app.toolbar UI testing using Robot Framework and Selenium

Timo Stollenwerk wrote:

> Am 08.09.2012 10:20, schrieb Asko Soukka:
>> Timo Stollenwerk wrote:
>>> Did anybody work on or look into the integration of robot framework
>>> into
>>> plone.app.testing recently?
>>
>> I and Godefroid looked into it a few months ago and came up with
>> different solutions.
> Awesome! I didn't know we made so much progress lately. Can't wait to
> look into it!
>
> Do you already have experience with running those tests on Jenkins?

Not really, but

'robotsuite'-based suites should run/report similarly to any existing
zope.testrunner tests

'plone.act'/'pybot'-approach has its own plugin
https://wiki.jenkins-ci.org/display/JENKINS/Robot+Framework+Plugin

-Asko

------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and
threat landscape has changed and how IT managers can respond. Discussions
will include endpoint security, mobile security and the latest in malware
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
_______________________________________________
Plone-developers mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/plone-developers
Martin Aspeli Martin Aspeli
Reply | Threaded
Open this post in threaded view
|

Re: plone.app.toolbar UI testing using Robot Framework and Selenium

In reply to this post by Timo Stollenwerk


On 8 September 2012 08:11, Timo Stollenwerk <[hidden email]> wrote:

Did anybody work on or look into the integration of robot framework into
plone.app.testing recently?

Yes. There was a thread on this (started by me, contributed to mainly by Asko and Godefroid) recently, and Asko blogged about one approach: http://datakurre.pandala.org/2012/09/getting-started-with-robotframework-and.html
 
The bottom line in my opinion is that we desperately need a way to test
our code with selenium (not only for deco) and we need it NOW. If we can
make this work with robot, great! If not we should choose whatever works
for us right now.

My suggestion is that we:

 - use Robotframework to drive Selenium
 - build a package of Robotframework keywords that do common Ploney things and abstract them away so they can survive e.g. upgrades of Plone
 - start by using Asko's approach (see blog post above)

I know Godefroid (and, I'm sure, others) has another point of view on the last point. I think we don't really know what's going to work best, but ultimately it doesn't really matter: 90% of what you do (write Robotframework tests, write keywords, write test layers setting up your stuff) is the same regardless of the two approaches. We should just get started, and I vote for getting started with the thing that is closest to what we do currently with all our other tests, and evolve that if it causes pain.

Martin

------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and
threat landscape has changed and how IT managers can respond. Discussions
will include endpoint security, mobile security and the latest in malware
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
_______________________________________________
Plone-developers mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/plone-developers
Ed Manlove-2 Ed Manlove-2
Reply | Threaded
Open this post in threaded view
|

Re: plone.app.toolbar UI testing using Robot Framework and Selenium

In reply to this post by Asko Soukka

> My solution was to wrap RF tests into Python unittest suites similarly
> to how doctests have been wrapped (that became 'robotsuite'):
>
> http://datakurre.pandala.org/2012/09/getting-started-with-robotframework-and.html
>

This is a great writeup Asko! Anyone interested in testing the UI of
their own product or Plone add-on should read this article.

Ed

------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and
threat landscape has changed and how IT managers can respond. Discussions
will include endpoint security, mobile security and the latest in malware
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
_______________________________________________
Plone-developers mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/plone-developers
Ed Manlove-2 Ed Manlove-2
Reply | Threaded
Open this post in threaded view
|

Re: plone.app.toolbar UI testing using Robot Framework and Selenium

In reply to this post by Asko Soukka
On 09/08/2012 05:22 AM, Asko Soukka wrote:

> Timo Stollenwerk wrote:
>> Am 08.09.2012 10:20, schrieb Asko Soukka:
>>> Timo Stollenwerk wrote:
>>>> Did anybody work on or look into the integration of robot framework
>>>> into
>>>> plone.app.testing recently?
>>> I and Godefroid looked into it a few months ago and came up with
>>> different solutions.
>> Awesome! I didn't know we made so much progress lately. Can't wait to
>> look into it!
>>
>> Do you already have experience with running those tests on Jenkins?
> Not really, but
>
> 'robotsuite'-based suites should run/report similarly to any existing
> zope.testrunner tests
>
> 'plone.act'/'pybot'-approach has its own plugin
> https://wiki.jenkins-ci.org/display/JENKINS/Robot+Framework+Plugin
>

I would be interested in any notes people take in creating a
jenkins/selenium/robotframework test system. As part of my effort to
assist with the robotframework-selenium2library project I've been
contemplating putting jenkins on my local machine, just to try it out
and see how it can be used for robotframework-selenium2library. In
particular it would be helpful in that we need to test across OSes,
Browsers, Python version, not too unlike Plone but also across several
"flavors" of Python incluing, CPython, PyPy, and especially IronPython.

Ed

------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and
threat landscape has changed and how IT managers can respond. Discussions
will include endpoint security, mobile security and the latest in malware
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
_______________________________________________
Plone-developers mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/plone-developers
Timo Stollenwerk Timo Stollenwerk
Reply | Threaded
Open this post in threaded view
|

Re: plone.app.toolbar UI testing using Robot Framework and Selenium

In reply to this post by Asko Soukka
Asko,

wow! I'm really impressed! Great work and writeup! Why is your blog not
on planet Plone?

I wrote some initial tests for p.a.collection and run those tests on
Jenkins:

https://jenkins.plone.org/job/plone.app.collection/

Seems like we can just use the robot framework Jenkins plugin to show
the results:

https://jenkins.plone.org/job/plone.app.collection/9/robot/?

There are still things I need to figure out, and if we want to run those
tests on a regular basis I guess we need more hardware and find a way to
run the tests in parallel, but this really looks great so far!

Cheers,
timo

ps: you are going to be at ploneconf this year right?

Am 08.09.2012 11:22, schrieb Asko Soukka:

> Timo Stollenwerk wrote:
>> Am 08.09.2012 10:20, schrieb Asko Soukka:
>>> Timo Stollenwerk wrote:
>>>> Did anybody work on or look into the integration of robot framework
>>>> into
>>>> plone.app.testing recently?
>>>
>>> I and Godefroid looked into it a few months ago and came up with
>>> different solutions.
>> Awesome! I didn't know we made so much progress lately. Can't wait to
>> look into it!
>>
>> Do you already have experience with running those tests on Jenkins?
>
> Not really, but
>
> 'robotsuite'-based suites should run/report similarly to any existing
> zope.testrunner tests
>
> 'plone.act'/'pybot'-approach has its own plugin
> https://wiki.jenkins-ci.org/display/JENKINS/Robot+Framework+Plugin
>
> -Asko


------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and
threat landscape has changed and how IT managers can respond. Discussions
will include endpoint security, mobile security and the latest in malware
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
_______________________________________________
Plone-developers mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/plone-developers
Timo Stollenwerk Timo Stollenwerk
Reply | Threaded
Open this post in threaded view
|

Re: plone.app.toolbar UI testing using Robot Framework and Selenium

In reply to this post by Martin Aspeli
Am 08.09.2012 12:45, schrieb Martin Aspeli:

>
>
> On 8 September 2012 08:11, Timo Stollenwerk <[hidden email]
> <mailto:[hidden email]>> wrote:
>
>
>     Did anybody work on or look into the integration of robot framework into
>     plone.app.testing recently?
>
>
> Yes. There was a thread on this (started by me, contributed to mainly by
> Asko and Godefroid) recently, and Asko blogged about one approach:
> http://datakurre.pandala.org/2012/09/getting-started-with-robotframework-and.html

Yeah. I saw that thread. You guys lost me at some point though.

>     The bottom line in my opinion is that we desperately need a way to test
>     our code with selenium (not only for deco) and we need it NOW. If we can
>     make this work with robot, great! If not we should choose whatever works
>     for us right now.
>
>
> My suggestion is that we:
>
>   - use Robotframework to drive Selenium
>   - build a package of Robotframework keywords that do common Ploney
> things and abstract them away so they can survive e.g. upgrades of Plone
>   - start by using Asko's approach (see blog post above)

Am I correct in assuming that with Godefroid's approach we currently
have to test against a real instance and we don't have test isolation
and proper setUp/tearDown? If this is true I'm all for Asko's approach.
As said before, we need something that works NOW.

We will have to write lots of tests for deco during the sea sprint and I
don't want to invest too much time into struggeling with test
infrastructure.

Actually I would like us to make a decision asap and then start to work
on that approach before sea sprint. I will have some time to spare next
week and I'm happy to work on whatever approach we can agree on.

> I know Godefroid (and, I'm sure, others) has another point of view on
> the last point. I think we don't really know what's going to work best,
> but ultimately it doesn't really matter: 90% of what you do (write
> Robotframework tests, write keywords, write test layers setting up your
> stuff) is the same regardless of the two approaches. We should just get
> started, and I vote for getting started with the thing that is closest
> to what we do currently with all our other tests, and evolve that if it
> causes pain.

Exactly. If it turns out that Godefroid's approach is better, we can
always switch with rather low costs.

@Godefroid: Could you maybe elaborate your approach again and tell us
something about the current status?

Cheers,
timo

------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and
threat landscape has changed and how IT managers can respond. Discussions
will include endpoint security, mobile security and the latest in malware
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
_______________________________________________
Plone-developers mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/plone-developers
Asko Soukka Asko Soukka
Reply | Threaded
Open this post in threaded view
|

Re: plone.app.toolbar UI testing using Robot Framework and Selenium

Timo Stollenwerk wrote:
> Am I correct in assuming that with Godefroid's approach we currently
> have to test against a real instance and we don't have test isolation
> and proper setUp/tearDown?
No, both ways should give you the same test isolation.

In my way, the extra glue code is in 'robotsuite', in wrapping RF test
suites into Python unittest compatible test suites, and tests could be
run with zope.testrunner. In Godefroid's way, the extra glue code is in
custom RF keywords to start Zope2 server and run
plone.app.testing-layers on top of it, and test are run with pure pybot.

AFAIK, eventually both ways could use the same testing layers and, if
Godefroid would adopt the same http port as plone.testing for
Zope-server, it should be trivial to swap between these approaches. And
neither approach should prevent you from running the tests also against
"real instance" (to speed up the tests while you are still in development).

-Asko

------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and
threat landscape has changed and how IT managers can respond. Discussions
will include endpoint security, mobile security and the latest in malware
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
_______________________________________________
Plone-developers mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/plone-developers
Martin Aspeli Martin Aspeli
Reply | Threaded
Open this post in threaded view
|

Re: plone.app.toolbar UI testing using Robot Framework and Selenium



On 9 September 2012 15:49, Asko Soukka <[hidden email]> wrote:
Timo Stollenwerk wrote:
Am I correct in assuming that with Godefroid's approach we currently have to test against a real instance and we don't have test isolation and proper setUp/tearDown?
No, both ways should give you the same test isolation.

In my way, the extra glue code is in 'robotsuite', in wrapping RF test suites into Python unittest compatible test suites, and tests could be run with zope.testrunner. In Godefroid's way, the extra glue code is in custom RF keywords to start Zope2 server and run plone.app.testing-layers on top of it, and test are run with pure pybot.

To put it differently:

 - Asko's approach is more like other tests we write for Plone
 - Godefroid's approach is more like other tests you'd write with Robotframework
 
My gut feel is that the lowest barrier to entry for existing Plone developers will be the former, since it fits into our toolchain and expectations about how tests are run. I'm not seeing a lot of people *already* familiar with Robotframework who want to write Plone tests and are confused by plone.app.testing (and furthermore, you still need to know how to write a layer to do any kind of non-trivial testing of specific packages even with Godefroid's approach).

There is a risk that integration (i.e. Asko's robotsuite package) proves difficult to maintain or buggy. So far, it seems really tiny, though. It's not quite perfect (some of the reporting is a bit off), but from what I could see so far I'm not hugely worried about it. That's why I'd prefer to start there.

AFAIK, eventually both ways could use the same testing layers and, if Godefroid would adopt the same http port as plone.testing for Zope-server, it should be trivial to swap between these approaches. And neither approach should prevent you from running the tests also against "real instance" (to speed up the tests while you are still in development).

Agreed. This means the cost of reversing our starting position is relatively low if it turns out to be wrong.

Martin 

------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and
threat landscape has changed and how IT managers can respond. Discussions
will include endpoint security, mobile security and the latest in malware
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
_______________________________________________
Plone-developers mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/plone-developers
Asko Soukka Asko Soukka
Reply | Threaded
Open this post in threaded view
|

Re: plone.app.toolbar UI testing using Robot Framework and Selenium

Martin Aspeli wrote:
> There is a risk that integration (i.e. Asko's robotsuite package)
> proves difficult to maintain or buggy. So far, it seems really tiny,
> though. It's not quite perfect (some of the reporting is a bit off),
> but from what I could see so far I'm not hugely worried about it.
> That's why I'd prefer to start there.

It's under Collective from today https://github.com/collective/robotsuite
and if you'd prefer it to be owned by Plone, I don't have anything
against it.

-Asko

------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and
threat landscape has changed and how IT managers can respond. Discussions
will include endpoint security, mobile security and the latest in malware
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
_______________________________________________
Plone-developers mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/plone-developers
Godefroid Chapelle Godefroid Chapelle
Reply | Threaded
Open this post in threaded view
|

Re: plone.app.toolbar UI testing using Robot Framework and Selenium

In reply to this post by Asko Soukka
Le 09/09/12 16:49, Asko Soukka a écrit :
> AFAIK, eventually both ways could use the same testing layers and, if
> Godefroid would adopt the same http port as plone.testing for
> Zope-server, it should be trivial to swap between these approaches. And
> neither approach should prevent you from running the tests also against
> "real instance" (to speed up the tests while you are still in development).
>
> -Asko

I wonder if trying to have Jenkins run tests suite with the two
approaches would not help to ensure that the suites and keywords are
modular and independent enough of Plone implementation details.

IMO, this is the most important goal, test keywords that are DRY and
hide implementation details as much as possible.

Regards
--
Godefroid Chapelle (aka __gotcha) http://bubblenet.be


------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and
threat landscape has changed and how IT managers can respond. Discussions
will include endpoint security, mobile security and the latest in malware
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
_______________________________________________
Plone-developers mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/plone-developers
Dylan Jay Dylan Jay
Reply | Threaded
Open this post in threaded view
|

Re: plone.app.toolbar UI testing using Robot Framework and Selenium

In reply to this post by Asko Soukka
On 10/09/2012, at 2:07 AM, Asko Soukka wrote:

> Martin Aspeli wrote:
>> There is a risk that integration (i.e. Asko's robotsuite package)
>> proves difficult to maintain or buggy. So far, it seems really tiny,
>> though. It's not quite perfect (some of the reporting is a bit off),
>> but from what I could see so far I'm not hugely worried about it.
>> That's why I'd prefer to start there.
>
> It's under Collective from today https://github.com/collective/robotsuite
> and if you'd prefer it to be owned by Plone, I don't have anything
> against it.

I don't want to slow down momentum as there seems to be a consensus on  
the approach which is great.

I just wanted to ask if there is a way to use the robot BDD syntax  
without selenium?
I'd like to start using this with some of my existing plugins but I  
don't use any JS so something based on mechanize (like testbrowser) is  
enough and would run faster I'm guessing (as less has to be loaded up  
to run a test).


>
> -Asko
>
> ------------------------------------------------------------------------------
> Live Security Virtual Conference
> Exclusive live event will cover all the ways today's security and
> threat landscape has changed and how IT managers can respond.  
> Discussions
> will include endpoint security, mobile security and the latest in  
> malware
> threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
> _______________________________________________
> Plone-developers mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/plone-developers


------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and
threat landscape has changed and how IT managers can respond. Discussions
will include endpoint security, mobile security and the latest in malware
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
_______________________________________________
Plone-developers mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/plone-developers
Asko Soukka Asko Soukka
Reply | Threaded
Open this post in threaded view
|

Re: plone.app.toolbar UI testing using Robot Framework and Selenium

Dylan Jay wrote:
On 10/09/2012, at 2:07 AM, Asko Soukka wrote:
> I just wanted to ask if there is a way to use the robot BDD syntax
> without selenium?

Sure there is.

Robot Framework is not tied to Selenium in any way. You could write your
robot tests in BDD syntax (only in English, though) and connect
the tests to your code by writing the actual testkeywords in Python in
your product. (With Selenium, we are just using keywords written in
selenium2library.)

http://robotframework.googlecode.com/hg/doc/userguide/RobotFrameworkUserGuide.html?r=2.7.4#behavior-driven-style

http://robotframework.googlecode.com/hg/doc/userguide/RobotFrameworkUserGuide.html?r=2.7.4#embedding-arguments-into-keyword-name

See also, how "Keywords"-class is defined and used in:

http://datakurre.pandala.org/2012/09/getting-started-with-robotframework-and.html#loggingin

Also, there's still Martin's Python-version of CoreJet for writing
BDD-style tests directly for Python unittest:

http://pypi.python.org/pypi/corejet.core

In CoreJet, there's no implicit ways to re-use keywords and therefore
Robot gets you closer to, e.g. cucumber. On the other hand, CoreJet's
other idea was to extract your stories from some issue tracking system.
I've used it with Pivotal Tracker with

http://pypi.python.org/pypi/corejet.pivotal

And, actually, also generated robot tests from the extracted data with

http://pypi.python.org/pypi/corejet.robot

But, after a year, I haven't managed to convince our team to try out
CoreJet toolchain enough, to know, if it's worth of the extra effort.

-Asko

------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and
threat landscape has changed and how IT managers can respond. Discussions
will include endpoint security, mobile security and the latest in malware
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
_______________________________________________
Plone-developers mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/plone-developers
Dylan Jay Dylan Jay
Reply | Threaded
Open this post in threaded view
|

Re: plone.app.toolbar UI testing using Robot Framework and Selenium


On 10/09/2012, at 2:04 PM, Asko Soukka wrote:

> Dylan Jay wrote:
> On 10/09/2012, at 2:07 AM, Asko Soukka wrote:
>> I just wanted to ask if there is a way to use the robot BDD syntax
>> without selenium?
>
> Sure there is.
>
> Robot Framework is not tied to Selenium in any way. You could write  
> your robot tests in BDD syntax (only in English, though) and connect
> the tests to your code by writing the actual testkeywords in Python  
> in your product. (With Selenium, we are just using keywords written  
> in selenium2library.)
>
> http://robotframework.googlecode.com/hg/doc/userguide/RobotFrameworkUserGuide.html?r=2.7.4#behavior-driven-style
>
> http://robotframework.googlecode.com/hg/doc/userguide/RobotFrameworkUserGuide.html?r=2.7.4#embedding-arguments-into-keyword-name
>
> See also, how "Keywords"-class is defined and used in:
>
> http://datakurre.pandala.org/2012/09/getting-started-with-robotframework-and.html#loggingin

ok. I think I understand.

So if I wanted to swap from using non-selenium to using selenium  
easily (say if I add JS UI later to my code) I'd have to ensure that  
all my keywords have two sets of implementations. That isn't handled  
by a lower level?
The example I'm thinking of is the branch of zope.testbrowser called  
zope.testbrowser.real which let you use the same tests but ran them in  
a real browser instead without having to rewrite anything  (see http://svn.zope.org/zope.testbrowser/branches/benji-zope.testbrowser.real/src/zope/testbrowser/real.py)

So the way to achieve this with robot is to always use the same plone  
keywords so we can switch out one implementation for another?

Something I don't yet understand is why robot over another BDD  
framework like behave or lettuce etc? Is it that it already comes with  
support for selenium? or is it just the reusable keywords feature?

http://packages.python.org/behave/tutorial.html

>
> Also, there's still Martin's Python-version of CoreJet for writing  
> BDD-style tests directly for Python unittest:
>
> http://pypi.python.org/pypi/corejet.core
>
> In CoreJet, there's no implicit ways to re-use keywords and therefore
> Robot gets you closer to, e.g. cucumber. On the other hand,  
> CoreJet's other idea was to extract your stories from some issue  
> tracking system. I've used it with Pivotal Tracker with
>
> http://pypi.python.org/pypi/corejet.pivotal
>
> And, actually, also generated robot tests from the extracted data with
>
> http://pypi.python.org/pypi/corejet.robot
>
> But, after a year, I haven't managed to convince our team to try out  
> CoreJet toolchain enough, to know, if it's worth of the extra effort.
>
> -Asko


------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and
threat landscape has changed and how IT managers can respond. Discussions
will include endpoint security, mobile security and the latest in malware
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
_______________________________________________
Plone-developers mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/plone-developers
Timo Stollenwerk Timo Stollenwerk
Reply | Threaded
Open this post in threaded view
|

Re: plone.app.toolbar UI testing using Robot Framework and Selenium

In reply to this post by Godefroid Chapelle
Am 09.09.2012 23:12, schrieb Godefroid Chapelle:

> Le 09/09/12 16:49, Asko Soukka a écrit :
>> AFAIK, eventually both ways could use the same testing layers and, if
>> Godefroid would adopt the same http port as plone.testing for
>> Zope-server, it should be trivial to swap between these approaches. And
>> neither approach should prevent you from running the tests also against
>> "real instance" (to speed up the tests while you are still in development).
>>
>> -Asko
>
> I wonder if trying to have Jenkins run tests suite with the two
> approaches would not help to ensure that the suites and keywords are
> modular and independent enough of Plone implementation details.

I would like to give your approach another try to get a better
understanding of the differences. Can you give me a pointer to the most
recent buildout/howto?

Can we move plone.act to the collective or plone core repo so people can
start to contribute?

I'd also like to write some documentation about robot keywords as well
as our (plone.act) keywords. Did anybody started to write documentation
yet? Where shall we put it?

Cheers,
Timo

------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and
threat landscape has changed and how IT managers can respond. Discussions
will include endpoint security, mobile security and the latest in malware
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
_______________________________________________
Plone-developers mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/plone-developers
Asko Soukka Asko Soukka
Reply | Threaded
Open this post in threaded view
|

Re: plone.app.toolbar UI testing using Robot Framework and Selenium

In reply to this post by Dylan Jay
Dylan Jay wrote:

>> Dylan Jay wrote:
>> On 10/09/2012, at 2:07 AM, Asko Soukka wrote:
>>> I just wanted to ask if there is a way to use the robot BDD syntax
>>> without selenium?
>> Sure there is.
>>
>> Robot Framework is not tied to Selenium in any way. You could write
>> your robot tests in BDD syntax (only in English, though) and connect
>> the tests to your code by writing the actual testkeywords in Python
>> in your product. (With Selenium, we are just using keywords written
>> in selenium2library.)
>>
>> http://robotframework.googlecode.com/hg/doc/userguide/RobotFrameworkUserGuide.html?r=2.7.4#behavior-driven-style 
>>
>>
>> http://robotframework.googlecode.com/hg/doc/userguide/RobotFrameworkUserGuide.html?r=2.7.4#embedding-arguments-into-keyword-name 
>>
>>
>> See also, how "Keywords"-class is defined and used in:
>>
>> http://datakurre.pandala.org/2012/09/getting-started-with-robotframework-and.html#loggingin 
>>
>
> ok. I think I understand.
>
> So if I wanted to swap from using non-selenium to using selenium
> easily (say if I add JS UI later to my code) I'd have to ensure that
> all my keywords have two sets of implementations. That isn't handled
> by a lower level?

Yes. Easy "swapping" would require us to have a custom Robot Framework
keyword library, which implements as much of
robotframework.selenium2library with zope.testrunner as required. If we
had one, just importing a different keyword library would make the swap.
Because selenium2library is also written in Python, you could use its
keyword-modules as skeletons for the new library.

> Something I don't yet understand is why robot over another BDD
> framework like behave or lettuce etc? Is it that it already comes with
> support for selenium? or is it just the reusable keywords feature?
I'd say both, and the p.a.testing integrations we now have. A few things
Robot is missing from BDD point of view, are localized BDD-keywords
(Given, When, Then) and example tables (e.g. in cucumber/behave/lettuce).

-Asko

------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and
threat landscape has changed and how IT managers can respond. Discussions
will include endpoint security, mobile security and the latest in malware
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
_______________________________________________
Plone-developers mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/plone-developers
12