Java SMF automated database backup

[mitch2k]

Hey,

backup works from the admin interface, but not when I turn on GZIP compression. Strange, cause this worked before.

The SMF DB is on an apache shared server. Is there anyway to force your program not to use gzip?

Thanks alot!

[viulian]

If it worked before, it means that probably your database grew in size, and gzip-ing it takes a lot of memory (hence the internal server error as probably the php script is killed when it reaches an upper memory limit).

I cannot disable the gzip compression - the reason I choose not to implement it was that if there's not enough memory for gzip, probably the database is big - and transferring it unzipped will do a lot of traffic on your account.

If you need it quick (I am in process of rethinking the code and implement the changes other people asked for) and do not want to work on the existing code.

Do you know how to compile a java program ? The source code is available, and it's easy to compile it (I'll explain what to change to turn off the automated gzip).

[mitch2k]

Well, it worked before, and the database is only 12MB big... Wich should not be a problem I guess.

Maybe it is a temporary problem with my host (one.com).

I have no experience with Java programming. But I'll take a look at the Sun tutorials, to if it's hard to learn...

Thanks!!

[res6cue]

Do you know how to compile a java program ? The source code is available, and it's easy to compile it (I'll explain what to change to turn off the automated gzip).

Please advise what steps I have to take to turn gzip off.  This is a great program, but my database is just too large for the server's upper memory limit to handle compressing it before download.  Transfer bandwidth usage is of no concern to me, but having a daily, automated backup of my database is!

[viulian]

If you got to the point where you can compile the application - you'll find the tutorial inside the source code - you can then:

1) Open the file smfbackup.http.SMFConnection.java
2) Find the line:

Code Select Expand

currentMethod = new GetMethod(config.getProperty("forum.url") + "?struct=on&data=on&compress=gzip&action=dumpdb&sesc=" + sesc);
3) remove this part

Code Select Expand

&compress=gzip
(make sure that an & remains between data=on&action=dumpdb)
4) Recompile and that's it

[res6cue]

Thanks.  Now the question is:  Where do I find the source files I'll need to recompile?!?

[viulian]

The address was listed on the first page of the topic.
Please use your favorite SVN client and download them from:

http://javasmfbackup.googlecode.com/svn/releases/v1.01/

To compile, please check the COMPILE.txt file in there, you will need some utilities which are mentioned in there..

[res6cue]

Mission accomplished!

I'm not sure why you won't publish a non-compressed version of this program, or have a compression on/off option per server.  It was almost a deal killer for me, as my database is too large to have compression enabled due to the limited upper memory my host has set.  It's a fantastic program otherwise, but there are a lot of people out there running into this same issue who have no earthly clue how to recompile programs.

I should point out that your instructions were incomplete, for those who might attempt to turn off compression and recompile themselves.  The following 3 lines in src\smfbackup\http\SMFConnection.java need to be changed in order to disable compression and name the file properly:

Line 254 changed from:

Code Select Expand

currentMethod = new GetMethod(config.getProperty("forum.url") + "?struct=on&data=on&compress=gzip&action=dumpdb&sesc=" + sesc);

To:

Code Select Expand

currentMethod = new GetMethod(config.getProperty("forum.url") + "?struct=on&data=on&action=dumpdb&sesc=" + sesc);


Line 259 changed from:

Code Select Expand

forum.updateStatus("Proceeding to backup. GET on " + config.getProperty("forum.url") + "?struct=on&data=on&compress=gzip&action=dumpdb&sesc=" + sesc);

To:

Code Select Expand

forum.updateStatus("Proceeding to backup. GET on " + config.getProperty("forum.url") + "?struct=on&data=on&action=dumpdb&sesc=" + sesc);


Line 307 from:

Code Select Expand

String path = Configuration.getBackupDir() + System.getProperty("file.separator") + forum.getName() + "_" + sdf.format(new Date()) + ".sql.gz";

To:

Code Select Expand

String path = Configuration.getBackupDir() + System.getProperty("file.separator") + forum.getName() + "_" + sdf.format(new Date()) + ".sql";

[res6cue]

One other issue you need to fix is that the program gives an exception if Administration Security is enabled in SMF (which it is by default):

class smfbackup.http.SMFException: No "sesc" or "sc" input field found in the reply page. Cannot proceed.

I suspect it has something to do with this bit of code in SMFConnection.java:

Code Select Expand

        currentMethod.releaseConnection();
       
        if (sesc == null)
            throw new SMFException("No \"sesc\" or \"sc\" input field found in the reply page. Cannot proceed.");


We should not have to disable the Administration Security option in SMF just for the sake of running a backup.  It's an unnecessary security risk to do so.  There should be no reason your program can't send the password again when admin.php asks for it!

[viulian]

Yah, thanks for pointing those lines out I coded this software an year ago or so - so I forgot.

About the Administrative Security issue, I've quickly glanced over the php code and it seems not as straight forward as you put it.
Basically sesc / sc should be there when the admin user tries to access the administrative pages. If admin's session has more than 1 hour inactivity, then the login prompt will popup [which happens unless you have checked "Disable Administrative Security"].

This means that software is able to run at 2AM when the prev. session already expired, and is able to find out the sesc/sc parameter to make SMF understand that's the same user.
Also, this was not an error that was reported - and should have popped out a lot, since it makes the program futile . So I'm thinking that maybe it's something particular on your server.. probably related to the admin session time or something.

I want to release a new version, but too many things to do - and too little time I hope to release it in one month or so..

Thanks again for the info, if you have other things, please put them here, I will read the whole thread again.. when I start to code.

[viulian]

I've just done some test runs with an updated version, which works with SMF 2.0 Beta 3.

I feel I need to redo parts of the code - to handle the new SMF version (mainly different action URLs + another way of detecting the session code + the actual backup action is now a POST instead of a GET).
Also, UI will need a face lift as well as adding the gzip option, per the request above.

Also, current version of the application has a bug because it assumes the SMF Cookie (which is named in Settings.php) is always named "SMFCookie". I will need to handle this case automatically so this will be a bug fix.

I need to add some more logging, so when errors appear, they will be more meaningful.

So I am working on it

[metallica48423]

Good to hear

[scalawaggirl]

Viulian, I could just kiss you!  Thanks so much for sharing this.  The SMF backups have been the bane of my existence for too long and you've saved me!  I've had two disasters with SMF, so relevant back ups are well appreciated.

Looking forward to the new version and enhancements.  Great stuff.

Thanks!!!!!

[viulian]

I've just finished version 1.2b with support for SMF 2.0 Beta 3/3.1. [Attention, these version is not the latest anymore. Check the webpage for the latest, and use that please.]

Here's the download page: http://hex.ro/wb/pages/projects/java-smf-backup.php

ATTENTION: Before uninstalling, please make a copy of smfbackup.conf file found in the application directory. It will work with the new version, and will save you the time to redefine the configuration you have.

Change Log:

  * Added support for SMF 2.0 Beta 3 / 3.1.
  * Added download counter (indicator) for each forum (to the bottom left of the main screen when backup is running).
  * Compression can now be turned on/off for each forum.
  * Face lift - Silk icons + draggable tool bar.
  * Made the logs more user friendly.
  * Improved exception logging in the log file
  * Added Tips And Tricks dialog.
  * Added Change Log dialog.

I'm looking forward to your feedback

[Sudhakar Arjunan]

will try now and give you a feedback immediately.

[scalawaggirl]

I just uninstalled my old version to reinstall this one (install worked fine); however, I am getting the follow error when trying to run a backup:

[09/06 13:30:01] java.lang.NullPointerException

   at smfbackup.http.SMFConnection.doLogin(Unknown Source)

   at smfbackup.http.SMFConnection.doBackup(Unknown Source)

   at smfbackup.http.SMFConnection.doJob(Unknown Source)

   at hxxp:smfbackup.run.manageable.run [nonactive](Unknown Source)

I am running SMF 1.1.5.

[viulian]

Ahh... I forgot to add the debug info parameter to the ANT script, and instead of some useful line numbers I see (Unknown source).

a) Did you use the previous configuration file, or did you recreate your configuration ?

I'm creating a new installer with the application having debug enabled and will reply.

[scalawaggirl]

Hi, I saved it originally but got a weird error, so started fresh.  This error is w/the fresh config.

[Sudhakar Arjunan]

Hi , i have tried many time,

1st time uninstalled previous version, then reinstalled

2nd time installed a fresh copy by clearing all files

3rd time renamed the existing folder and installed a new copy.

Everytime i have only this error.

Connecting to admin backend.

java.lang.StringIndexOutOfBoundsException: String index out of range: -1

   at java.lang.String.substring(Unknown Source)

   at smfbackup.http.SMFCookieContainer.trimCookiePathAndDomain(Unknown Source)

   at smfbackup.http.SMFCookieContainer.saveCookies(Unknown Source)

   at smfbackup.http.SMFConnection.doLogin(Unknown Source)

   at smfbackup.http.SMFConnection.doBackup(Unknown Source)

   at smfbackup.http.SMFConnection.doJob(Unknown Source)

   at smfbackup.run.Manageable.run(Unknown Source)


Using 1.1.5, sys win xp.

Kindly help me out.

[viulian]

I've deployed the new versions (Windows and Linux). Warning! these are not the latest anymore, please check the my web page for the latest version

@scalawaggirl - unfortunately I cannot say what the error is, but the new version will print the line numbers instead of (Unknown source) and that will pinpoint the error location easier. Please let me know.

@asudhakar -> Fixed the error.

For other errors - please enable Debug logging mode (in Settings) and send me a PM with the smfbackup.log file (password isn't displayed there, and if you do not want your forum name to be seen, you can paste it first in Notepad and search and replace the name).