Quantcast

backup of blobstorage in collective.recipe.backup

classic Classic list List threaded Threaded
10 messages Options
Vincent Fretin Vincent Fretin
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

backup of blobstorage in collective.recipe.backup

Hi,

Do somebody plan to implement the backup of the blobstorage in
collective.recipe.backup?
I don't have the time to look into that myself. For now I use my hacky
script which tar.gz the var/blobstorage.
I have a cronjob which execute bin/backup.sh every night.

------------------------------------------------------
[buildout]
extends = buildout.cfg
parts +=
    repozo
    backup
    backup-template
    backup-schedule

[repozo]
recipe = zc.recipe.egg
eggs = ZODB3
scripts = repozo

[backup]
recipe = collective.recipe.backup
keep = 7
full = true
gzip = true

[backup-template]
recipe = collective.recipe.template
inline =
    #!/bin/bash
    ${buildout:bin-directory}/zeopack
    ${buildout:bin-directory}/backup -q
    blob=$(basename `ls ${backup:location}|tail -n1` .fsz)
    cd ${buildout:directory}
    tar zcf ${backup:location}/$blob.tar.gz var/blobstorage
    toremove=`ls ${backup:location}/*.tar.gz|sort -r|tail -n+8`
    if [ ! -z "$toremove"]; then rm $toremove; fi
    rsync -a --delete ${backup:location}/ user@host:/home/user/extranet/backups/
    rsync -a --delete ${buildout:directory}/var/log/
user@host:/home/user/extranet/log/
output = ${buildout:bin-directory}/backup.sh
mode = 755

[backup-schedule]
recipe = z3c.recipe.usercrontab
times = 0 3 * * *
command = ${backup-template:output}
------------------------------------------------------

which produce in var/backups/
2010-08-09-01-00-06.dat
2010-08-09-01-00-06.fsz
2010-08-09-01-00-06.tar.gz


and to restore, I use
bin/restore
rm -rf var/blobstorage/*
tar xvf var/backups/2010-08-09-01-00-06.tar.gz

Vincent Fretin
Ecreall
Site : http://vincentfretin.ecreall.com

------------------------------------------------------------------------------
This SF.net email is sponsored by

Make an app they can't live without
Enter the BlackBerry Developer Challenge
http://p.sf.net/sfu/RIM-dev2dev 
_______________________________________________
Plone-developers mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/plone-developers
Maurits van Rees-3 Maurits van Rees-3
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: backup of blobstorage in collective.recipe.backup

Op 11-08-10 09:44, Vincent Fretin schreef:
> Hi,
>
> Do somebody plan to implement the backup of the blobstorage in
> collective.recipe.backup?

Not yet.  I have just added a comment on trunk to warn about this.  Code
for the recipe is here:
http://svn.plone.org/svn/collective/buildout/collective.recipe.backup/trunk

So for now, something manual is needed, like you do below; I'm looking
forward to a cross platform patch from someone. :-)

> I don't have the time to look into that myself. For now I use my hacky
> script which tar.gz the var/blobstorage.
> I have a cronjob which execute bin/backup.sh every night.
>
> ------------------------------------------------------
> [buildout]
> extends = buildout.cfg
> parts +=
>      repozo
>      backup
>      backup-template
>      backup-schedule
>
> [repozo]
> recipe = zc.recipe.egg
> eggs = ZODB3
> scripts = repozo
>
> [backup]
> recipe = collective.recipe.backup
> keep = 7
> full = true
> gzip = true

Note: 'gzip = true' is the default since version 0.8 (so since almost
two years).

> [backup-template]
> recipe = collective.recipe.template
> inline =
>      #!/bin/bash
>      ${buildout:bin-directory}/zeopack
>      ${buildout:bin-directory}/backup -q
>      blob=$(basename `ls ${backup:location}|tail -n1` .fsz)
>      cd ${buildout:directory}
>      tar zcf ${backup:location}/$blob.tar.gz var/blobstorage
>      toremove=`ls ${backup:location}/*.tar.gz|sort -r|tail -n+8`
>      if [ ! -z "$toremove"]; then rm $toremove; fi
>      rsync -a --delete ${backup:location}/ user@host:/home/user/extranet/backups/
>      rsync -a --delete ${buildout:directory}/var/log/
> user@host:/home/user/extranet/log/
> output = ${buildout:bin-directory}/backup.sh
> mode = 755
>
> [backup-schedule]
> recipe = z3c.recipe.usercrontab
> times = 0 3 * * *
> command = ${backup-template:output}
> ------------------------------------------------------
>
> which produce in var/backups/
> 2010-08-09-01-00-06.dat
> 2010-08-09-01-00-06.fsz
> 2010-08-09-01-00-06.tar.gz

This may take up quite a lot of space, especially when you backup
multiple times a day.  repozo uses incremental backups (unless you have
full=true like you have) and stores a deltafs; best would be to have
something like a delta tarball as well.  I don't know how doable that
is, especially in a cross platform way.  But I have not tried it.

I have not played around much with blobstorage yet, so I don't know the
pitfalls with regard to consistency here.

Does anyone know if e.g. a zeopack also throws away old versions of
files in blobstorage?

> and to restore, I use
> bin/restore
> rm -rf var/blobstorage/*
> tar xvf var/backups/2010-08-09-01-00-06.tar.gz
>
> Vincent Fretin

Oh, thanks for the lines you posted a while ago to get a bin/repozo
script in your buildout if it is missing; I have added it to the README
and made a release 1.4 a few days ago so it shows up on pypi.

--
Maurits van Rees
Programmer, Zest Software


------------------------------------------------------------------------------
This SF.net email is sponsored by

Make an app they can't live without
Enter the BlackBerry Developer Challenge
http://p.sf.net/sfu/RIM-dev2dev 
_______________________________________________
Plone-developers mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/plone-developers
Vincent Fretin Vincent Fretin
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: backup of blobstorage in collective.recipe.backup

On Wed, Aug 11, 2010 at 10:25 AM, Maurits van Rees
<[hidden email]> wrote:
> Does anyone know if e.g. a zeopack also throws away old versions of
> files in blobstorage?

I tested it, bin/zeopack packs the blobstorage as well, yes.

I tested the versionning of ATFile, it works too.
It creates 3 blobs by version, I don't know why.
So initially when you create an ATFile, you have 4 blobs.
If you reupload the file, you get 8 blobs, and if you pack, you get 7 blobs.

Vincent

------------------------------------------------------------------------------
This SF.net email is sponsored by

Make an app they can't live without
Enter the BlackBerry Developer Challenge
http://p.sf.net/sfu/RIM-dev2dev 
_______________________________________________
Plone-developers mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/plone-developers
AnthonyG AnthonyG
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: backup of blobstorage in collective.recipe.backup

> It creates 3 blobs by version, I don't know why.

Something to do with image scales?

_______________________________________

Anthony Gerrard
Email: [hidden email]
http://www.sponsorachild.org.uk

On 11/08/2010 12:12, Vincent Fretin wrote:

> On Wed, Aug 11, 2010 at 10:25 AM, Maurits van Rees
> <[hidden email]>  wrote:
>> Does anyone know if e.g. a zeopack also throws away old versions of
>> files in blobstorage?
>
> I tested it, bin/zeopack packs the blobstorage as well, yes.
>
> I tested the versionning of ATFile, it works too.
> It creates 3 blobs by version, I don't know why.
> So initially when you create an ATFile, you have 4 blobs.
> If you reupload the file, you get 8 blobs, and if you pack, you get 7 blobs.
>
> Vincent
>

------------------------------------------------------------------------------
This SF.net email is sponsored by

Make an app they can't live without
Enter the BlackBerry Developer Challenge
http://p.sf.net/sfu/RIM-dev2dev 
_______________________________________________
Plone-developers mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/plone-developers
Vincent Fretin Vincent Fretin
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: backup of blobstorage in collective.recipe.backup

On Wed, Aug 11, 2010 at 1:20 PM, Anthony Gerrard
<[hidden email]> wrote:
>> It creates 3 blobs by version, I don't know why.
>
> Something to do with image scales?

Image scales for a simple ATFile?

If the versionning is disabled, you only have 1 blob when you create the ATFile.
If you overwrite the file, you get 1 blob more, and so on.
But with versionning, you get 4 blobs each time, the normal 1, +3
blobs in CMFEditions storage.

Vincent

------------------------------------------------------------------------------
This SF.net email is sponsored by

Make an app they can't live without
Enter the BlackBerry Developer Challenge
http://p.sf.net/sfu/RIM-dev2dev 
_______________________________________________
Plone-developers mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/plone-developers
Nejc Zupan Nejc Zupan
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: backup of blobstorage in collective.recipe.backup

In reply to this post by Maurits van Rees-3
On 11.8.2010, at 10:25, Maurits van Rees wrote:

Op 11-08-10 09:44, Vincent Fretin schreef:
Hi,

Do somebody plan to implement the backup of the blobstorage in
collective.recipe.backup?

Not yet.  I have just added a comment on trunk to warn about this.  Code
for the recipe is here:
http://svn.plone.org/svn/collective/buildout/collective.recipe.backup/trunk

So for now, something manual is needed, like you do below; I'm looking
forward to a cross platform patch from someone. :-)


Oh, I really want to see this happen! I've been trying to work it out somehow,
but my knowledge is far to limited here to come up with a solid solution. 

I want to help out in any way I can to get this done ASAP, 
either writing documentation, tests or doing actual user testing. 



Best regards,
Nejc Zupan

www.niteoweb.com -- NiteoWeb, web development and consulting, Ltd.


------------------------------------------------------------------------------
This SF.net email is sponsored by

Make an app they can't live without
Enter the BlackBerry Developer Challenge
http://p.sf.net/sfu/RIM-dev2dev 
_______________________________________________
Plone-developers mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/plone-developers
Ramon Navarro Bosch Ramon Navarro Bosch
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: backup of blobstorage in collective.recipe.backup

In reply to this post by Maurits van Rees-3
Hi every body, 

We developed a script to allow incremental backups on blob storage. 

We do a full dump once a month and incremental copy evey day. For the DB we use packing and repoze. For the blobs we create a copy of the blobs using hard links ( they need to be on the same partition ) to a current-blobs directory and then using rsync to maintain for every day on the month the incremental copy. This allow that at every day folder we have all the blobs of each day but they don't consume the amount of days x (size of blob storage). For example folder 01 (means the blobs backup of first day of the month) have 29Gbytes of blob storage and the folder that contain all days blob backup its 35Gbytes (instead of the 29Gbytes x 31 days) On our case we also have different fs and blob storage for each plone ( we are nearly 300 ).

Hope to help and inspire for ideas! I'm sending the backup script also, is GPL so do whatever you want!

Ramon



2010/8/11 Maurits van Rees <[hidden email]>
Op 11-08-10 09:44, Vincent Fretin schreef:
> Hi,
>
> Do somebody plan to implement the backup of the blobstorage in
> collective.recipe.backup?

Not yet.  I have just added a comment on trunk to warn about this.  Code
for the recipe is here:
http://svn.plone.org/svn/collective/buildout/collective.recipe.backup/trunk

So for now, something manual is needed, like you do below; I'm looking
forward to a cross platform patch from someone. :-)

> I don't have the time to look into that myself. For now I use my hacky
> script which tar.gz the var/blobstorage.
> I have a cronjob which execute bin/backup.sh every night.
>
> ------------------------------------------------------
> [buildout]
> extends = buildout.cfg
> parts +=
>      repozo
>      backup
>      backup-template
>      backup-schedule
>
> [repozo]
> recipe = zc.recipe.egg
> eggs = ZODB3
> scripts = repozo
>
> [backup]
> recipe = collective.recipe.backup
> keep = 7
> full = true
> gzip = true

Note: 'gzip = true' is the default since version 0.8 (so since almost
two years).

> [backup-template]
> recipe = collective.recipe.template
> inline =
>      #!/bin/bash
>      ${buildout:bin-directory}/zeopack
>      ${buildout:bin-directory}/backup -q
>      blob=$(basename `ls ${backup:location}|tail -n1` .fsz)
>      cd ${buildout:directory}
>      tar zcf ${backup:location}/$blob.tar.gz var/blobstorage
>      toremove=`ls ${backup:location}/*.tar.gz|sort -r|tail -n+8`
>      if [ ! -z "$toremove"]; then rm $toremove; fi
>      rsync -a --delete ${backup:location}/ user@host:/home/user/extranet/backups/
>      rsync -a --delete ${buildout:directory}/var/log/
> user@host:/home/user/extranet/log/
> output = ${buildout:bin-directory}/backup.sh
> mode = 755
>
> [backup-schedule]
> recipe = z3c.recipe.usercrontab
> times = 0 3 * * *
> command = ${backup-template:output}
> ------------------------------------------------------
>
> which produce in var/backups/
> 2010-08-09-01-00-06.dat
> 2010-08-09-01-00-06.fsz
> 2010-08-09-01-00-06.tar.gz

This may take up quite a lot of space, especially when you backup
multiple times a day.  repozo uses incremental backups (unless you have
full=true like you have) and stores a deltafs; best would be to have
something like a delta tarball as well.  I don't know how doable that
is, especially in a cross platform way.  But I have not tried it.

I have not played around much with blobstorage yet, so I don't know the
pitfalls with regard to consistency here.

Does anyone know if e.g. a zeopack also throws away old versions of
files in blobstorage?

> and to restore, I use
> bin/restore
> rm -rf var/blobstorage/*
> tar xvf var/backups/2010-08-09-01-00-06.tar.gz
>
> Vincent Fretin

Oh, thanks for the lines you posted a while ago to get a bin/repozo
script in your buildout if it is missing; I have added it to the README
and made a release 1.4 a few days ago so it shows up on pypi.

--
Maurits van Rees
Programmer, Zest Software


------------------------------------------------------------------------------
This SF.net email is sponsored by

Make an app they can't live without
Enter the BlackBerry Developer Challenge
http://p.sf.net/sfu/RIM-dev2dev
_______________________________________________
Plone-developers mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/plone-developers



--
Ramon a.k.a bloodbare

------------------------------------------------------------------------------
This SF.net email is sponsored by

Make an app they can't live without
Enter the BlackBerry Developer Challenge
http://p.sf.net/sfu/RIM-dev2dev 
_______________________________________________
Plone-developers mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/plone-developers

backup.sh (2K) Download Attachment
Vincent Fretin Vincent Fretin
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: backup of blobstorage in collective.recipe.backup

Hi,

Thanks Ramon for the script. The idea of hard links is nice, but your
backups have to be on the same partition. You need to have minimum
raid1 on your server. ;)

I like to have the backups on another server. I've done a quick
search, you can do an incremental backup with tar easily like this:
start a full backup on day 0:
tar cf bak0.tar var/blobstorage
on day 1, do an incremental backup, it backups only newer file:
find var/blobstorage ! -type d -newer bak0.tar -print | xargs tar cf bak0.1.tar
on day 2:
find var/blobstorage ! -type d -newer bak0.tar -print | xargs tar cf bak0.2.tar

To restore the blobstorage in the state of day2:
rm -rf var/blobstorage/*
tar xf bak0.tar
tar xf bak0.2.tar

Vincent Fretin
Ecreall
Site : http://vincentfretin.ecreall.com



On Fri, Aug 13, 2010 at 12:08 PM, Ramon Navarro Bosch
<[hidden email]> wrote:

> Hi every body,
> We developed a script to allow incremental backups on blob storage.
> We do a full dump once a month and incremental copy evey day. For the DB we
> use packing and repoze. For the blobs we create a copy of the blobs using
> hard links ( they need to be on the same partition ) to a current-blobs
> directory and then using rsync to maintain for every day on the month the
> incremental copy. This allow that at every day folder we have all the blobs
> of each day but they don't consume the amount of days x (size of blob
> storage). For example folder 01 (means the blobs backup of first day of the
> month) have 29Gbytes of blob storage and the folder that contain all days
> blob backup its 35Gbytes (instead of the 29Gbytes x 31 days) On our case we
> also have different fs and blob storage for each plone ( we are nearly 300
> ).
> Hope to help and inspire for ideas! I'm sending the backup script also, is
> GPL so do whatever you want!
> Ramon
>
>
> 2010/8/11 Maurits van Rees <[hidden email]>
>>
>> Op 11-08-10 09:44, Vincent Fretin schreef:
>> > Hi,
>> >
>> > Do somebody plan to implement the backup of the blobstorage in
>> > collective.recipe.backup?
>>
>> Not yet.  I have just added a comment on trunk to warn about this.  Code
>> for the recipe is here:
>>
>> http://svn.plone.org/svn/collective/buildout/collective.recipe.backup/trunk
>>
>> So for now, something manual is needed, like you do below; I'm looking
>> forward to a cross platform patch from someone. :-)
>>
>> > I don't have the time to look into that myself. For now I use my hacky
>> > script which tar.gz the var/blobstorage.
>> > I have a cronjob which execute bin/backup.sh every night.
>> >
>> > ------------------------------------------------------
>> > [buildout]
>> > extends = buildout.cfg
>> > parts +=
>> >      repozo
>> >      backup
>> >      backup-template
>> >      backup-schedule
>> >
>> > [repozo]
>> > recipe = zc.recipe.egg
>> > eggs = ZODB3
>> > scripts = repozo
>> >
>> > [backup]
>> > recipe = collective.recipe.backup
>> > keep = 7
>> > full = true
>> > gzip = true
>>
>> Note: 'gzip = true' is the default since version 0.8 (so since almost
>> two years).
>>
>> > [backup-template]
>> > recipe = collective.recipe.template
>> > inline =
>> >      #!/bin/bash
>> >      ${buildout:bin-directory}/zeopack
>> >      ${buildout:bin-directory}/backup -q
>> >      blob=$(basename `ls ${backup:location}|tail -n1` .fsz)
>> >      cd ${buildout:directory}
>> >      tar zcf ${backup:location}/$blob.tar.gz var/blobstorage
>> >      toremove=`ls ${backup:location}/*.tar.gz|sort -r|tail -n+8`
>> >      if [ ! -z "$toremove"]; then rm $toremove; fi
>> >      rsync -a --delete ${backup:location}/
>> > user@host:/home/user/extranet/backups/
>> >      rsync -a --delete ${buildout:directory}/var/log/
>> > user@host:/home/user/extranet/log/
>> > output = ${buildout:bin-directory}/backup.sh
>> > mode = 755
>> >
>> > [backup-schedule]
>> > recipe = z3c.recipe.usercrontab
>> > times = 0 3 * * *
>> > command = ${backup-template:output}
>> > ------------------------------------------------------
>> >
>> > which produce in var/backups/
>> > 2010-08-09-01-00-06.dat
>> > 2010-08-09-01-00-06.fsz
>> > 2010-08-09-01-00-06.tar.gz
>>
>> This may take up quite a lot of space, especially when you backup
>> multiple times a day.  repozo uses incremental backups (unless you have
>> full=true like you have) and stores a deltafs; best would be to have
>> something like a delta tarball as well.  I don't know how doable that
>> is, especially in a cross platform way.  But I have not tried it.
>>
>> I have not played around much with blobstorage yet, so I don't know the
>> pitfalls with regard to consistency here.
>>
>> Does anyone know if e.g. a zeopack also throws away old versions of
>> files in blobstorage?
>>
>> > and to restore, I use
>> > bin/restore
>> > rm -rf var/blobstorage/*
>> > tar xvf var/backups/2010-08-09-01-00-06.tar.gz
>> >
>> > Vincent Fretin
>>
>> Oh, thanks for the lines you posted a while ago to get a bin/repozo
>> script in your buildout if it is missing; I have added it to the README
>> and made a release 1.4 a few days ago so it shows up on pypi.
>>
>> --
>> Maurits van Rees
>> Programmer, Zest Software
>>
>>
>>
>> ------------------------------------------------------------------------------
>> This SF.net email is sponsored by
>>
>> Make an app they can't live without
>> Enter the BlackBerry Developer Challenge
>> http://p.sf.net/sfu/RIM-dev2dev
>> _______________________________________________
>> Plone-developers mailing list
>> [hidden email]
>> https://lists.sourceforge.net/lists/listinfo/plone-developers
>
>
>
> --
> Ramon a.k.a bloodbare
>
> ------------------------------------------------------------------------------
> This SF.net email is sponsored by
>
> Make an app they can't live without
> Enter the BlackBerry Developer Challenge
> http://p.sf.net/sfu/RIM-dev2dev
> _______________________________________________
> Plone-developers mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/plone-developers
>
>

------------------------------------------------------------------------------
This SF.net email is sponsored by

Make an app they can't live without
Enter the BlackBerry Developer Challenge
http://p.sf.net/sfu/RIM-dev2dev 
_______________________________________________
Plone-developers mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/plone-developers
Andreas Zeidler Andreas Zeidler
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: backup of blobstorage in collective.recipe.backup

In reply to this post by Vincent Fretin
On 11.08.10 13:32, Vincent Fretin wrote:

> On Wed, Aug 11, 2010 at 1:20 PM, Anthony Gerrard
> <[hidden email]> wrote:
>>> It creates 3 blobs by version, I don't know why.
>>
>> Something to do with image scales?
>
> Image scales for a simple ATFile?
>
> If the versionning is disabled, you only have 1 blob when you create the ATFile.
> If you overwrite the file, you get 1 blob more, and so on.
> But with versionning, you get 4 blobs each time, the normal 1, +3
> blobs in CMFEditions storage.

are any of them empty by chance?


andi

--
zeidler it consulting - http://zitc.de/ - [hidden email]
friedelstraße 31 - 12047 berlin - telefon +49 30 25563779
pgp key at http://zitc.de/pgp - http://wwwkeys.de.pgp.net/
plone 4.0 released! -- http://plone.org/4


------------------------------------------------------------------------------
This SF.net Dev2Dev email is sponsored by:

Show off your parallel programming skills.
Enter the Intel(R) Threading Challenge 2010.
http://p.sf.net/sfu/intel-thread-sfd
_______________________________________________
Plone-developers mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/plone-developers
Vincent Fretin Vincent Fretin
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: backup of blobstorage in collective.recipe.backup

On Tue, Sep 7, 2010 at 9:52 AM, Andreas Zeidler <[hidden email]> wrote:

> On 11.08.10 13:32, Vincent Fretin wrote:
>> On Wed, Aug 11, 2010 at 1:20 PM, Anthony Gerrard
>> <[hidden email]> wrote:
>>>> It creates 3 blobs by version, I don't know why.
>>>
>>> Something to do with image scales?
>>
>> Image scales for a simple ATFile?
>>
>> If the versionning is disabled, you only have 1 blob when you create the ATFile.
>> If you overwrite the file, you get 1 blob more, and so on.
>> But with versionning, you get 4 blobs each time, the normal 1, +3
>> blobs in CMFEditions storage.
>
> are any of them empty by chance?

Hi Andreas,

I just tested with latest plone-coredev.
I got one new blob + 2 empty blobs for each revision.

Vincent

------------------------------------------------------------------------------
Start uncovering the many advantages of virtual appliances
and start using them to simplify application deployment and
accelerate your shift to cloud computing
http://p.sf.net/sfu/novell-sfdev2dev
_______________________________________________
Plone-developers mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/plone-developers
Loading...