cut or copy and paste "large" plone site?

classic Classic list List threaded Threaded
6 messages Options
John Schinnerer John Schinnerer
Reply | Threaded
Open this post in threaded view
|

cut or copy and paste "large" plone site?

Hi,

I think this is the place for install/upgrade issues. If not please
direct me to the appropriate place.

Plone 3.3.5 is the platform here, but the reason for the question is to
facilitate upgrading a 3.3.5 site to 4.x.

Is it possible to cut/copy and paste a "large" plone site, within a
given zope instance that has one or more existing plone sites?

Also, is there a way to verify that "clear and rebuild" and "update" of
catalog (portal_catalog -> Advanced tab) complete, or not, for a "large"
site?

"Large" in this case means a Data.fs of ~5.7GB.

I am asking because I have started "clear and rebuild" and "update"
operations multiple times, but after some long time of the browser
"waiting for domain.tld...," the right pane of the ZMI refreshes with a
"connection refused."

I am getting the same result when I try and copy or cut and paste the
site. In that case it seems clear the operation failed at some point,
since the site remains in the original location when I refresh the ZMI
tab (was not moved or copied).

Does one of the logs track these operations? Or is there any other way
to find out what is actually happening, especially with the catalog
operations?

thanks,
John S.

--
John Schinnerer - M.A., Whole Systems Design
--------------------------------------------
- Eco-Living -
Whole Systems Design Services
People - Place - Learning - Integration
[hidden email] - 510.982.1334
http://eco-living.net
http://sociocracyconsulting.com
_______________________________________________
Setup mailing list
[hidden email]
https://lists.plone.org/mailman/listinfo/plone-setup
Steve McMahon Steve McMahon
Reply | Threaded
Open this post in threaded view
|

Re: cut or copy and paste "large" plone site?

You should find an event.log file in your client directories. Sometimes put in a separate log directory.

You may wish to start a client in foreground mode and directly address it while doing this kind of experimenting.

Watching the log or foreground console will give you a pretty good idea of whether or not the clear/rebuild is reindexing. If your site is not just large in bulk, but has a lot of content objects, you'll need to enable subtransactions.

Other strategies:

* Set up a mirror of the old site (using backup data) and simply delete everything but the target site. If you want to combine it others at the end of the update process, you may use mount points.

* Use Zope's export/import facility, restoring to a mirror of the old setup (same versions/add-ons). If you restore to a different folder, be ready to rebuild the catalog.

On Tue, Dec 15, 2015 at 10:59 AM, John Schinnerer <[hidden email]> wrote:
Hi,

I think this is the place for install/upgrade issues. If not please direct me to the appropriate place.

Plone 3.3.5 is the platform here, but the reason for the question is to facilitate upgrading a 3.3.5 site to 4.x.

Is it possible to cut/copy and paste a "large" plone site, within a given zope instance that has one or more existing plone sites?

Also, is there a way to verify that "clear and rebuild" and "update" of catalog (portal_catalog -> Advanced tab) complete, or not, for a "large" site?

"Large" in this case means a Data.fs of ~5.7GB.

I am asking because I have started "clear and rebuild" and "update" operations multiple times, but after some long time of the browser "waiting for domain.tld...," the right pane of the ZMI refreshes with a "connection refused."

I am getting the same result when I try and copy or cut and paste the site. In that case it seems clear the operation failed at some point, since the site remains in the original location when I refresh the ZMI tab (was not moved or copied).

Does one of the logs track these operations? Or is there any other way to find out what is actually happening, especially with the catalog operations?

thanks,
John S.

--
John Schinnerer - M.A., Whole Systems Design
--------------------------------------------
- Eco-Living -
Whole Systems Design Services
People - Place - Learning - Integration
[hidden email] - <a href="tel:510.982.1334" value="+15109821334" target="_blank">510.982.1334
http://eco-living.net
http://sociocracyconsulting.com
_______________________________________________
Setup mailing list
[hidden email]
https://lists.plone.org/mailman/listinfo/plone-setup


_______________________________________________
Setup mailing list
[hidden email]
https://lists.plone.org/mailman/listinfo/plone-setup
John Schinnerer John Schinnerer
Reply | Threaded
Open this post in threaded view
|

Re: cut or copy and paste "large" plone site?

Thanks Steve, very helpful.
Additional info/questions for the list:

On 12/15/2015 11:19 AM, Steve McMahon wrote:
> You should find an event.log file in your client directories. Sometimes
> put in a separate log directory.

Yep, found them. I tried both cut/paste and copy/paste of the site.

Log shows that the cut/paste attempt apparently ended after a whole lot
of these errors (too many open files):
--------
2015-12-15T08:31:35 INFO Archetypes Error while trying to convert file
contents to 'text/plain' in <Field file(file:rw)>.getIndexable() of
<ATFile at
/copy_of_hawaiicountycdp.info/hilo/bayfront-trails/Public_Meeting_2
_Presentation-Press_Quality.pdf>: [Errno 24] Too many open files:
'/tmp/tmpktmYI0'
--------

The copy/paste ended with this error, in the ZMI and detailed in the log:
----
Error Type: ImportError
Error Value: No module named collective.recaptcha.settings
--------------

...which I have also gotten trying to import the same site into a
different location in the same instance, after exporting it successfully.
So it looks like copy/paste does an export/import behind the scenes,
while cut/paste does...something else...?

Re the ImportError - the 3.3.5 site is an upgrade from a 3.2.2 site that
used collective.recaptcha. In that upgrade process, I found that adding
collective.recaptcha to the 3.3.5 buildout caused an error on every page
render attempt. It was used on only two forms, not mission critical at
all, so I took it back out of the 3.3.5 buildout. The two forms then
threw errors rendering, but after I deleted the recaptcha objects from
the form folders they worked again.
I thought that was the end of it, but apparently there is some kind of
collective.recaptcha ghost in the machine, as it is still trying to
import a module.
No idea what to do about that, suggestions welcome.

> Watching the log or foreground console will give you a pretty good idea
> of whether or not the clear/rebuild is reindexing. If your site is not
> just large in bulk, but has a lot of content objects, you'll need to
> enable subtransactions.

Subtransactions are enabled.
The threshold is set at 10000 (not by me - some kind of default?) - the
catalog has 9039 records - so I assume that means it will do the whole
catalog in one transaction, unless I lower the threshold?

I will post separately w/new subject line for the rest of this since it
leads to what seems to be the core issue (cut/copy & paste the site
being an attempt to work around that issue that is not working).

thanks,
John S.

> On Tue, Dec 15, 2015 at 10:59 AM, John Schinnerer <[hidden email]
> <mailto:[hidden email]>> wrote:
>
>     Hi,
>
>     I think this is the place for install/upgrade issues. If not please
>     direct me to the appropriate place.
>
>     Plone 3.3.5 is the platform here, but the reason for the question is
>     to facilitate upgrading a 3.3.5 site to 4.x.
>
>     Is it possible to cut/copy and paste a "large" plone site, within a
>     given zope instance that has one or more existing plone sites?
>
>     Also, is there a way to verify that "clear and rebuild" and "update"
>     of catalog (portal_catalog -> Advanced tab) complete, or not, for a
>     "large" site?
>
>     "Large" in this case means a Data.fs of ~5.7GB.
>
>     I am asking because I have started "clear and rebuild" and "update"
>     operations multiple times, but after some long time of the browser
>     "waiting for domain.tld...," the right pane of the ZMI refreshes
>     with a "connection refused."
>
>     I am getting the same result when I try and copy or cut and paste
>     the site. In that case it seems clear the operation failed at some
>     point, since the site remains in the original location when I
>     refresh the ZMI tab (was not moved or copied).
>
>     Does one of the logs track these operations? Or is there any other
>     way to find out what is actually happening, especially with the
>     catalog operations?
>
>     thanks,
>     John S.
>
>     --
>     John Schinnerer - M.A., Whole Systems Design
>     --------------------------------------------
>     - Eco-Living -
>     Whole Systems Design Services
>     People - Place - Learning - Integration
>     [hidden email] <mailto:[hidden email]> - 510.982.1334
>     <tel:510.982.1334>
>     http://eco-living.net
>     http://sociocracyconsulting.com
>     _______________________________________________
>     Setup mailing list
>     [hidden email] <mailto:[hidden email]>
>     https://lists.plone.org/mailman/listinfo/plone-setup
>
>
>
>
> _______________________________________________
> Setup mailing list
> [hidden email]
> https://lists.plone.org/mailman/listinfo/plone-setup
>

--
John Schinnerer - M.A., Whole Systems Design
--------------------------------------------
- Eco-Living -
Whole Systems Design Services
People - Place - Learning - Integration
[hidden email] - 510.982.1334
http://eco-living.net
http://sociocracyconsulting.com
_______________________________________________
Setup mailing list
[hidden email]
https://lists.plone.org/mailman/listinfo/plone-setup
espen espen
Reply | Threaded
Open this post in threaded view
|

Re: cut or copy and paste "large" plone site?

This post has NOT been accepted by the mailing list yet.
Warning: do not copy / paste between different mount points, especially if you use blobs.
I would duplicate the Data.fs and delete the others (as suggested in the other email).
If you really need to copy / paste the site, I would 
1) Export the site from the ZMI (as a ZEXP file)
2) rename the site in ZMI
3) Import the ZEXP file  (you will need to put in in /var/instance/import

This will give you errors on import if something is ‘wrong’ with the site (for example if some product has been removed or unproperly installed

Espen

15. des. 2015 kl. 22.48 skrev John Schinnerer [via Plone] <[hidden email]>:

Thanks Steve, very helpful.
Additional info/questions for the list:

On 12/15/2015 11:19 AM, Steve McMahon wrote:
> You should find an event.log file in your client directories. Sometimes
> put in a separate log directory.

Yep, found them. I tried both cut/paste and copy/paste of the site.

Log shows that the cut/paste attempt apparently ended after a whole lot
of these errors (too many open files):
--------
2015-12-15T08:31:35 INFO Archetypes Error while trying to convert file
contents to 'text/plain' in <Field file(file:rw)>.getIndexable() of
<ATFile at
/copy_of_hawaiicountycdp.info/hilo/bayfront-trails/Public_Meeting_2
_Presentation-Press_Quality.pdf>: [Errno 24] Too many open files:
'/tmp/tmpktmYI0'
--------

The copy/paste ended with this error, in the ZMI and detailed in the log:
----
Error Type: ImportError
Error Value: No module named collective.recaptcha.settings
--------------

...which I have also gotten trying to import the same site into a
different location in the same instance, after exporting it successfully.
So it looks like copy/paste does an export/import behind the scenes,
while cut/paste does...something else...?

Re the ImportError - the 3.3.5 site is an upgrade from a 3.2.2 site that
used collective.recaptcha. In that upgrade process, I found that adding
collective.recaptcha to the 3.3.5 buildout caused an error on every page
render attempt. It was used on only two forms, not mission critical at
all, so I took it back out of the 3.3.5 buildout. The two forms then
threw errors rendering, but after I deleted the recaptcha objects from
the form folders they worked again.
I thought that was the end of it, but apparently there is some kind of
collective.recaptcha ghost in the machine, as it is still trying to
import a module.
No idea what to do about that, suggestions welcome.

> Watching the log or foreground console will give you a pretty good idea
> of whether or not the clear/rebuild is reindexing. If your site is not
> just large in bulk, but has a lot of content objects, you'll need to
> enable subtransactions.

Subtransactions are enabled.
The threshold is set at 10000 (not by me - some kind of default?) - the
catalog has 9039 records - so I assume that means it will do the whole
catalog in one transaction, unless I lower the threshold?

I will post separately w/new subject line for the rest of this since it
leads to what seems to be the core issue (cut/copy & paste the site
being an attempt to work around that issue that is not working).

thanks,
John S.

> On Tue, Dec 15, 2015 at 10:59 AM, John Schinnerer <<a href="x-msg://5/user/SendEmail.jtp?type=node&amp;node=7574704&amp;i=0" target="_top" rel="nofollow" link="external" class="">[hidden email]
> <mailto:<a href="x-msg://5/user/SendEmail.jtp?type=node&amp;node=7574704&amp;i=1" target="_top" rel="nofollow" link="external" class="">[hidden email]>> wrote:
>
>     Hi,
>
>     I think this is the place for install/upgrade issues. If not please
>     direct me to the appropriate place.
>
>     Plone 3.3.5 is the platform here, but the reason for the question is
>     to facilitate upgrading a 3.3.5 site to 4.x.
>
>     Is it possible to cut/copy and paste a "large" plone site, within a
>     given zope instance that has one or more existing plone sites?
>
>     Also, is there a way to verify that "clear and rebuild" and "update"
>     of catalog (portal_catalog -> Advanced tab) complete, or not, for a
>     "large" site?
>
>     "Large" in this case means a Data.fs of ~5.7GB.
>
>     I am asking because I have started "clear and rebuild" and "update"
>     operations multiple times, but after some long time of the browser
>     "waiting for domain.tld...," the right pane of the ZMI refreshes
>     with a "connection refused."
>
>     I am getting the same result when I try and copy or cut and paste
>     the site. In that case it seems clear the operation failed at some
>     point, since the site remains in the original location when I
>     refresh the ZMI tab (was not moved or copied).
>
>     Does one of the logs track these operations? Or is there any other
>     way to find out what is actually happening, especially with the
>     catalog operations?
>
>     thanks,
>     John S.
>
>     --
>     John Schinnerer - M.A., Whole Systems Design
>     --------------------------------------------
>     - Eco-Living -
>     Whole Systems Design Services
>     People - Place - Learning - Integration
>     <a href="x-msg://5/user/SendEmail.jtp?type=node&amp;node=7574704&amp;i=2" target="_top" rel="nofollow" link="external" class="">[hidden email] <mailto:<a href="x-msg://5/user/SendEmail.jtp?type=node&amp;node=7574704&amp;i=3" target="_top" rel="nofollow" link="external" class="">[hidden email]> - 510.982.1334
>     <<a href="tel:510.982.1334" class="">tel:510.982.1334>
>     http://eco-living.net
>     http://sociocracyconsulting.com
>     _______________________________________________
>     Setup mailing list
>     <a href="x-msg://5/user/SendEmail.jtp?type=node&amp;node=7574704&amp;i=4" target="_top" rel="nofollow" link="external" class="">[hidden email] <mailto:<a href="x-msg://5/user/SendEmail.jtp?type=node&amp;node=7574704&amp;i=5" target="_top" rel="nofollow" link="external" class="">[hidden email]>
>     https://lists.plone.org/mailman/listinfo/plone-setup
>
>
>
>
> _______________________________________________
> Setup mailing list
> <a href="x-msg://5/user/SendEmail.jtp?type=node&amp;node=7574704&amp;i=6" target="_top" rel="nofollow" link="external" class="">[hidden email]
> https://lists.plone.org/mailman/listinfo/plone-setup
>
--
John Schinnerer - M.A., Whole Systems Design
--------------------------------------------
- Eco-Living -
Whole Systems Design Services
People - Place - Learning - Integration
<a href="x-msg://5/user/SendEmail.jtp?type=node&amp;node=7574704&amp;i=7" target="_top" rel="nofollow" link="external" class="">[hidden email] - 510.982.1334
http://eco-living.net
http://sociocracyconsulting.com
_______________________________________________
Setup mailing list
<a href="x-msg://5/user/SendEmail.jtp?type=node&amp;node=7574704&amp;i=8" target="_top" rel="nofollow" link="external" class="">[hidden email]
https://lists.plone.org/mailman/listinfo/plone-setup



If you reply to this email, your message will be added to the discussion below:
http://plone.293351.n2.nabble.com/cut-or-copy-and-paste-large-plone-site-tp7574702p7574704.html
To start a new topic under Installation, Setup, Upgrades, email [hidden email]
To unsubscribe from Plone, click here.
NAML

John Schinnerer John Schinnerer
Reply | Threaded
Open this post in threaded view
|

Re: keyError on upgrade, path/mount point issues (was: cut or copy and paste "large" plone site?)

In reply to this post by Steve McMahon
Hi again,

Here is what I was hoping to work around, possibly with cut/copy and
paste of the site.

On 12/15/2015 11:19 AM, Steve McMahon wrote:

> Other strategies:
>
> * Set up a mirror of the old site (using backup data) and simply delete
> everything but the target site. If you want to combine it others at the
> end of the update process, you may use mount points.

Not the problem (multiple sites in one Data.fs). It's just one site in
one Data.fs I'm dealing with here.

> * Use Zope's export/import facility, restoring to a mirror of the old
> setup (same versions/add-ons). If you restore to a different folder, be
> ready to rebuild the catalog.

Tried this also. The export works OK; import hits the ImportError I
mentioned in previous post - looking for something in
collective.recaptcha, removed in upgrade because in the 3.3.5 it causes
an error (with a huge mass of code output, not a typical orderly
traceback) on every page render attempt.

The bottom line upgrade problem, which I've failed to work around with
export/import or cut/copy-paste, is that when I put the 3.3.5 Data.fs
into the 4.0.8 instance and dry run the upgrade, I get a keyError on a
traversal. Specifically:

[lots of upgrade steps completing OK]
...
Ran upgrade step: Upgrade DCMI metadata storage in portal_metadata
Ran upgrade step: Add the recursive groups PAS plugin.
Upgrade aborted. Error:
Traceback (most recent call last):
File
"/opt/Plone-4.0.8-fresh/buildout-cache/eggs/Plone-4.0.8-py2.6.egg/Products/CMFPlone/MigrationTool.py",
line 175, in upgrade
step['step'].doStep(setup)
File
"/opt/Plone-4.0.8-fresh/buildout-cache/eggs/Products.GenericSetup-1.6.3-py2.6.egg/Products/GenericSetup/upgrade.py",
line 142, in doStep
self.handler(tool)
File
"/opt/Plone-4.0.8-fresh/buildout-cache/eggs/plone.app.upgrade-1.0.8-py2.6.egg/plone/app/upgrade/v40/alphas.py",
line 480, in updateLargeFolderType
update(brain)
File
"/opt/Plone-4.0.8-fresh/buildout-cache/eggs/plone.app.upgrade-1.0.8-py2.6.egg/plone/app/upgrade/v40/alphas.py",
line 473, in update
obj = brain.getObject()
File
"/opt/Plone-4.0.8-fresh/buildout-cache/eggs/Zope2-2.12.19-py2.6-linux-x86_64.egg/Products/ZCatalog/CatalogBrains.py",
line 83, in getObject
parent = parent.unrestrictedTraverse(path[:-1])
File
"/opt/Plone-4.0.8-fresh/buildout-cache/eggs/Zope2-2.12.19-py2.6-linux-x86_64.egg/OFS/Traversable.py",
line 293, in unrestrictedTraverse
raise e
KeyError: 'puna-cdp'

The most likely explanation I have found so far is in this thread:
http://sourceforge.net/p/plone/mailman/message/27276817/

Specifically the part about paths to objects not matching:
"The error you are getting is...happening while the migration script is
trying to find a particular piece of content...by moving through the
path elements leading to that object. 'KeyError'...is raised when
dictionary lookup on an object fails to find an expected key in the
dictionary...you used to have the site on a separate Data.fs...on a
mountpoint...my guess is that the ZODB has persisted path information
for that site and everything in it that includes the mountpoint...it may
be that the ZODB and its catalog think the site is still in the old
location and are looking for it there. Since it is not there, you get a
KeyError."

This Data.fs I am trying to upgrade has one migration/upgrade in its
history that I did not do. It was a server OS upgrade during which my
sysadmin did something while reinstalling/re-configuring that led to a
path issue.

Before, there was a ZODB mount point 'domain.tld' in the ZMI root folder
(configured in buildout as usual) and in that 'domain.tld' folder was
the plone site, like:
/domain.tld/plone-site

After, that somehow changed to have one more 'domain.tld' folder in the
path in the ZMI, like:
/domain.tld/domain.tld/plone-site

BUT on the file system (as configured in buildout all along) the mount
point was still just
domain.tld/Data.fs

In other words, when I copy this Data.fs into
domain.tld/Data.fs
on the file system, the result is
domain.tld/domain.tld/plone-site
in the ZMI.

I've tried to fix by export/import and recatalog, but import hits the
collective.recaptcha import error (with collective.recaptcha breaking
the 3.3.5 site if I add it). I tried leaving collective.recaptcha in and
dry-running upgrade, just to see what would happen. That fixed the
collective.recaptcha import error but then gave an ImportError on
registry.interfaces. No idea why or what to do with that.

Then I tried to fix by cut/copy paste to move the site and then
recatalog, with the copy/paste failing on import for same reasons as
above, and the cut/paste failing for reasons yet TBD.

I also tried to create the duplicate-directory mount point path in the
4.0.8 instance, without success.

If I have this in buildout:
---------------
[filestorage]
recipe = collective.recipe.filestorage
blob-storage = var/blobstorage-%(fs_part_name)s
parts =
     domain.tld
----------------

...it works just like in the 3.3.5 instance, namely, the filesystem
mount point path is
domain.tld/Data.fs

I tried changing it to

---------------
...
parts =
     domain.tld/domain.tld
------------------

...but that gave me a triplicate path on the file system!
In other words, when I went to add the mount point in the ZMI, it showed as
domain.tld/domain.tld/domain.tld

No idea why or if that's a bug (seems like one, but maybe it's not even
allowed to have a multi-part path for the 'parts' declaration, or the /
has to be escaped, or...?).
In any case, it didn't get me a duplicate of the internal path, so I
don't know if that would work if it did.

So...what to try next?
I am about out of ideas.
Or, I'm thinking of something rather over my head, like a script that
would go through the whole catalog and fix the path strangeness, remove
the duplicate path element, something like that (if that's even possible).

Any assistance appreciated.

thanks,
John S.


> On Tue, Dec 15, 2015 at 10:59 AM, John Schinnerer <[hidden email]
> <mailto:[hidden email]>> wrote:
>
>     Hi,
>
>     I think this is the place for install/upgrade issues. If not please
>     direct me to the appropriate place.
>
>     Plone 3.3.5 is the platform here, but the reason for the question is
>     to facilitate upgrading a 3.3.5 site to 4.x.
>
>     Is it possible to cut/copy and paste a "large" plone site, within a
>     given zope instance that has one or more existing plone sites?
>
>     Also, is there a way to verify that "clear and rebuild" and "update"
>     of catalog (portal_catalog -> Advanced tab) complete, or not, for a
>     "large" site?
>
>     "Large" in this case means a Data.fs of ~5.7GB.
>
>     I am asking because I have started "clear and rebuild" and "update"
>     operations multiple times, but after some long time of the browser
>     "waiting for domain.tld...," the right pane of the ZMI refreshes
>     with a "connection refused."
>
>     I am getting the same result when I try and copy or cut and paste
>     the site. In that case it seems clear the operation failed at some
>     point, since the site remains in the original location when I
>     refresh the ZMI tab (was not moved or copied).
>
>     Does one of the logs track these operations? Or is there any other
>     way to find out what is actually happening, especially with the
>     catalog operations?
>
>     thanks,
>     John S.
>
>     --
>     John Schinnerer - M.A., Whole Systems Design
>     --------------------------------------------
>     - Eco-Living -
>     Whole Systems Design Services
>     People - Place - Learning - Integration
>     [hidden email] <mailto:[hidden email]> - 510.982.1334
>     <tel:510.982.1334>
>     http://eco-living.net
>     http://sociocracyconsulting.com
>     _______________________________________________
>     Setup mailing list
>     [hidden email] <mailto:[hidden email]>
>     https://lists.plone.org/mailman/listinfo/plone-setup
>
>
>
>
> _______________________________________________
> Setup mailing list
> [hidden email]
> https://lists.plone.org/mailman/listinfo/plone-setup
>

--
John Schinnerer - M.A., Whole Systems Design
--------------------------------------------
- Eco-Living -
Whole Systems Design Services
People - Place - Learning - Integration
[hidden email] - 510.982.1334
http://eco-living.net
http://sociocracyconsulting.com
_______________________________________________
Setup mailing list
[hidden email]
https://lists.plone.org/mailman/listinfo/plone-setup
John Schinnerer John Schinnerer
Reply | Threaded
Open this post in threaded view
|

Re: keyError on upgrade, path/mount point issues

I've managed to get past this - posting my solution in case it helps
someone else somewhere somehow sometime.

By setting the sub-transactions threshold in the portal_catalog Advanced
tab to a relatively small number, I was able to get catalog update, and
catalog clear/rebuild, to finish successfully.

Default appeared to be 10000; I tried 1000 but it still failed with the
"too many files open" error in the log.
I just took it down to 100 for a next try and that worked. This was on a
test/dev install of the site so it didn't matter much if that setting
would have a negative effect on the production site. It is only for
getting through the upgrade tangles and challenges.

Once the catalog update and clear/rebuild were done, I copied the
Data.fs from the 3.3.5 site to the 4.0.8 site and the upgrade completed
successfully. No more keyError.

cheers,
John S.


On 12/15/2015 02:35 PM, John Schinnerer wrote:

> Hi again,
>
> Here is what I was hoping to work around, possibly with cut/copy and
> paste of the site.
>
> On 12/15/2015 11:19 AM, Steve McMahon wrote:
>
>> Other strategies:
>>
>> * Set up a mirror of the old site (using backup data) and simply delete
>> everything but the target site. If you want to combine it others at the
>> end of the update process, you may use mount points.
>
> Not the problem (multiple sites in one Data.fs). It's just one site in
> one Data.fs I'm dealing with here.
>
>> * Use Zope's export/import facility, restoring to a mirror of the old
>> setup (same versions/add-ons). If you restore to a different folder, be
>> ready to rebuild the catalog.
>
> Tried this also. The export works OK; import hits the ImportError I
> mentioned in previous post - looking for something in
> collective.recaptcha, removed in upgrade because in the 3.3.5 it causes
> an error (with a huge mass of code output, not a typical orderly
> traceback) on every page render attempt.
>
> The bottom line upgrade problem, which I've failed to work around with
> export/import or cut/copy-paste, is that when I put the 3.3.5 Data.fs
> into the 4.0.8 instance and dry run the upgrade, I get a keyError on a
> traversal. Specifically:
>
> [lots of upgrade steps completing OK]
> ...
> Ran upgrade step: Upgrade DCMI metadata storage in portal_metadata
> Ran upgrade step: Add the recursive groups PAS plugin.
> Upgrade aborted. Error:
> Traceback (most recent call last):
> File
> "/opt/Plone-4.0.8-fresh/buildout-cache/eggs/Plone-4.0.8-py2.6.egg/Products/CMFPlone/MigrationTool.py",
> line 175, in upgrade
> step['step'].doStep(setup)
> File
> "/opt/Plone-4.0.8-fresh/buildout-cache/eggs/Products.GenericSetup-1.6.3-py2.6.egg/Products/GenericSetup/upgrade.py",
> line 142, in doStep
> self.handler(tool)
> File
> "/opt/Plone-4.0.8-fresh/buildout-cache/eggs/plone.app.upgrade-1.0.8-py2.6.egg/plone/app/upgrade/v40/alphas.py",
> line 480, in updateLargeFolderType
> update(brain)
> File
> "/opt/Plone-4.0.8-fresh/buildout-cache/eggs/plone.app.upgrade-1.0.8-py2.6.egg/plone/app/upgrade/v40/alphas.py",
> line 473, in update
> obj = brain.getObject()
> File
> "/opt/Plone-4.0.8-fresh/buildout-cache/eggs/Zope2-2.12.19-py2.6-linux-x86_64.egg/Products/ZCatalog/CatalogBrains.py",
> line 83, in getObject
> parent = parent.unrestrictedTraverse(path[:-1])
> File
> "/opt/Plone-4.0.8-fresh/buildout-cache/eggs/Zope2-2.12.19-py2.6-linux-x86_64.egg/OFS/Traversable.py",
> line 293, in unrestrictedTraverse
> raise e
> KeyError: 'puna-cdp'
>
> The most likely explanation I have found so far is in this thread:
> http://sourceforge.net/p/plone/mailman/message/27276817/
>
> Specifically the part about paths to objects not matching:
> "The error you are getting is...happening while the migration script is
> trying to find a particular piece of content...by moving through the
> path elements leading to that object. 'KeyError'...is raised when
> dictionary lookup on an object fails to find an expected key in the
> dictionary...you used to have the site on a separate Data.fs...on a
> mountpoint...my guess is that the ZODB has persisted path information
> for that site and everything in it that includes the mountpoint...it may
> be that the ZODB and its catalog think the site is still in the old
> location and are looking for it there. Since it is not there, you get a
> KeyError."
>
> This Data.fs I am trying to upgrade has one migration/upgrade in its
> history that I did not do. It was a server OS upgrade during which my
> sysadmin did something while reinstalling/re-configuring that led to a
> path issue.
>
> Before, there was a ZODB mount point 'domain.tld' in the ZMI root folder
> (configured in buildout as usual) and in that 'domain.tld' folder was
> the plone site, like:
> /domain.tld/plone-site
>
> After, that somehow changed to have one more 'domain.tld' folder in the
> path in the ZMI, like:
> /domain.tld/domain.tld/plone-site
>
> BUT on the file system (as configured in buildout all along) the mount
> point was still just
> domain.tld/Data.fs
>
> In other words, when I copy this Data.fs into
> domain.tld/Data.fs
> on the file system, the result is
> domain.tld/domain.tld/plone-site
> in the ZMI.
>
> I've tried to fix by export/import and recatalog, but import hits the
> collective.recaptcha import error (with collective.recaptcha breaking
> the 3.3.5 site if I add it). I tried leaving collective.recaptcha in and
> dry-running upgrade, just to see what would happen. That fixed the
> collective.recaptcha import error but then gave an ImportError on
> registry.interfaces. No idea why or what to do with that.
>
> Then I tried to fix by cut/copy paste to move the site and then
> recatalog, with the copy/paste failing on import for same reasons as
> above, and the cut/paste failing for reasons yet TBD.
>
> I also tried to create the duplicate-directory mount point path in the
> 4.0.8 instance, without success.
>
> If I have this in buildout:
> ---------------
> [filestorage]
> recipe = collective.recipe.filestorage
> blob-storage = var/blobstorage-%(fs_part_name)s
> parts =
>      domain.tld
> ----------------
>
> ...it works just like in the 3.3.5 instance, namely, the filesystem
> mount point path is
> domain.tld/Data.fs
>
> I tried changing it to
>
> ---------------
> ...
> parts =
>      domain.tld/domain.tld
> ------------------
>
> ...but that gave me a triplicate path on the file system!
> In other words, when I went to add the mount point in the ZMI, it showed as
> domain.tld/domain.tld/domain.tld
>
> No idea why or if that's a bug (seems like one, but maybe it's not even
> allowed to have a multi-part path for the 'parts' declaration, or the /
> has to be escaped, or...?).
> In any case, it didn't get me a duplicate of the internal path, so I
> don't know if that would work if it did.
>
> So...what to try next?
> I am about out of ideas.
> Or, I'm thinking of something rather over my head, like a script that
> would go through the whole catalog and fix the path strangeness, remove
> the duplicate path element, something like that (if that's even possible).
>
> Any assistance appreciated.
>
> thanks,
> John S.
>
>
>> On Tue, Dec 15, 2015 at 10:59 AM, John Schinnerer <[hidden email]
>> <mailto:[hidden email]>> wrote:
>>
>>     Hi,
>>
>>     I think this is the place for install/upgrade issues. If not please
>>     direct me to the appropriate place.
>>
>>     Plone 3.3.5 is the platform here, but the reason for the question is
>>     to facilitate upgrading a 3.3.5 site to 4.x.
>>
>>     Is it possible to cut/copy and paste a "large" plone site, within a
>>     given zope instance that has one or more existing plone sites?
>>
>>     Also, is there a way to verify that "clear and rebuild" and "update"
>>     of catalog (portal_catalog -> Advanced tab) complete, or not, for a
>>     "large" site?
>>
>>     "Large" in this case means a Data.fs of ~5.7GB.
>>
>>     I am asking because I have started "clear and rebuild" and "update"
>>     operations multiple times, but after some long time of the browser
>>     "waiting for domain.tld...," the right pane of the ZMI refreshes
>>     with a "connection refused."
>>
>>     I am getting the same result when I try and copy or cut and paste
>>     the site. In that case it seems clear the operation failed at some
>>     point, since the site remains in the original location when I
>>     refresh the ZMI tab (was not moved or copied).
>>
>>     Does one of the logs track these operations? Or is there any other
>>     way to find out what is actually happening, especially with the
>>     catalog operations?
>>
>>     thanks,
>>     John S.
>>
>>     --
>>     John Schinnerer - M.A., Whole Systems Design
>>     --------------------------------------------
>>     - Eco-Living -
>>     Whole Systems Design Services
>>     People - Place - Learning - Integration
>>     [hidden email] <mailto:[hidden email]> - 510.982.1334
>>     <tel:510.982.1334>
>>     http://eco-living.net
>>     http://sociocracyconsulting.com
>>     _______________________________________________
>>     Setup mailing list
>>     [hidden email] <mailto:[hidden email]>
>>     https://lists.plone.org/mailman/listinfo/plone-setup
>>
>>
>>
>>
>> _______________________________________________
>> Setup mailing list
>> [hidden email]
>> https://lists.plone.org/mailman/listinfo/plone-setup
>>
>

--
John Schinnerer - M.A., Whole Systems Design
--------------------------------------------
- Eco-Living -
Whole Systems Design Services
People - Place - Learning - Integration
[hidden email] - 510.982.1334
http://eco-living.net
http://sociocracyconsulting.com
_______________________________________________
Setup mailing list
[hidden email]
https://lists.plone.org/mailman/listinfo/plone-setup