Advertisement:

Author Topic: How i mirror my Live VPS SMF (and some system files) to my Intranet Dev/Test  (Read 846 times)

Offline aegersz

  • SMF Hero
  • ******
  • Posts: 1,494
  • Gender: Male
  • "mods" junkie
    • dopetalk
CentOS6:

If anybody is interested, I have put a lot of effort into "mirroring" my live SMF VPS to my dev/test intranet (which is then propogated to hard drives and other intranet servers) so here are my Linux bash scripts and control files:

(runs on Live hourly and on-demand before mirroring) dbbacksmf.sh

Code: [Select]
echo "writing" /SQL/database_smf209_$(date '+%d-%m-%Y').sql "..."

mysqldump --opt -udbuser -pdbpass smf209 > /SQL/database_smf209_$(date '+%d-%m-%Y').sql

echo "Done !"

(runs from the Intranet -- main script) mirror.sh

Code: [Select]
# mirror live web system to the intranet

# rsync-live-html-with-delete
echo "Live HTML directories are now being copied (special) ..."
rsync -zpaAXPve ssh --delete --exclude-from=/data/exclusions-special root@forum.drugs-and-users.org:/var/www/html/ /rsync-live-html-with-delete/
echo

# rsync-live-html
echo "Live HTML directories are now being copied (normal) ..."
rsync -zpaAXPve ssh --exclude-from=/data/exclusions /rsync-live-html-with-delete/ /var/www/html/
rsync -zpaAXPve ssh --exclude-from=/data/exclusions root@forum.drugs-and-users.org:/var/www/html/ /var/www/html/
echo
scp forum:/var/www/html/private/index.html /var/www/html/private/
echo

# rsync-live-html-unique
echo "Unique HTML files are now being copied ..."
rsync -zpaAXPve ssh --include-from=/data/inclusions --exclude="*" root@forum.drugs-and-users.org:/var/www/html/  \
        /rsync-live-html-unique/
echo
#rsync -zpaAXPve ssh root@forum.drugs-and-users.org:/var/www/html/owncloud/config/config.php \
#       /rsync-live-html-unique/
#echo

# fix owndership
chown   -R apache:apache /var/www/html
echo

#dblivesmf
#
sh      dblivesmf.sh y
#       ------------ - <-- backup allDB ? (y/n)
#getlives
rsync-live-root
echo
rsync-live-bin
echo
rsync-live-etc
echo
rsync-live-git
echo

# getmore (enable when you wish)
#rsync -zpaAXPve ssh forum:/tips/ /tips/
#rsync -zpaAXPve ssh root@forum.drugs-and-users.org:/home/automailer/ /forum/automailer/
#rsync -zpaAXPve ssh root@forum.drugs-and-users.org:/home/ /forum/home/
#rsync -zpaAXPve ssh root@forum.drugs-and-users.org:/tmp/backup-config-manifests/ /forum/
#rsync -zpaAXPve ssh root@forum.drugs-and-users.org:/boot/ /forum/boot/
#rsync -zpaAXPve ssh root@forum.drugs-and-users.org:/var/log/ /forum/var/log/

(called by main) dblivesmf.sh

Code: [Select]
#
# mirror live web system to the intranet
#

diff=-3
bdate=$(date '+%d-%m-%Y' -d "$diff hours")
btime=$(date '+%d-%m-%Y_%H-%M-%S' -d "$diff hours")

echo    Copy database_smf209_$bdate.sql ...
rsync   -zpaAPve ssh root@forum.drugs-and-users.org:/SQL/database_smf209_$bdate.sql \
                /rsync-live-dbbackup
#               /forum/SQL/
echo
echo    Archive to database_smf209_$btime.sql ...
rsync   -paAPv /rsync-live-dbbackup /forum/SQL/database_smf209_$btime.sql
echo

if  [ "$1"  ==  "y" ]; then
        echo    Copy and archive database_allDBs_$bdate.sql ...
        rsync   -zpaAPve ssh root@forum.drugs-and-users.org:/SQL/database_allDBs_$bdate.sql \
                        /forum/SQL/database_allDBs_$bdate.sql
        echo
fi

# fix internal links
#       -----------------------
sh      db_links_convert_smf.sh
#       -----------------------

echo    Restoring the database visually with Pipe Viewer (very cool!) ...
echo
pv      /rsync-live-dbbackup-newlinks | mysql -udbuser -pdbpass smf209
echo
echo    Done so now run "repair_settings.php" ...
echo

(called by the above script) db_links_convert_smf.sh

Code: [Select]
#!/bin/bash

live="forum.drugs-and-users.org/index.php/topic"
dev="192.168.0.180/index.php/topic"

echo    Coverting all occurrences of $live to $dev ...
sed     "s|$live|$dev|g" /rsync-live-dbbackup > /rsync-live-dbbackup-newlinks
echo    "sed;"$?
echo

live="128.199.200.202"
dev="192.168.0.180"
echo    Coverting all occurrences of $live to $dev ...
sed -i  "s|$live|$dev|g" /rsync-live-dbbackup-newlinks
echo    "sed;"$?
echo

live="128.199.200.202"
dev="192.168.0.180"
echo    Coverting all occurrences of $live/private to $dev ...
sed -i  "s|$live|$dev|g" /var/www/html/private/index.html
echo    "sed;"$?
echo

live="128.199.200.202"
dev="192.168.0.180"
echo    Coverting all occurrences of $live/private5 to $dev ...
sed -i  "s|$live|$dev|g" /var/www/html/private5/index.html
echo    "sed;"$?
echo

(additional scripts and control files referenced by the main script)

rsync-live-etc
Code: [Select]
rsync -zpaAXPve ssh root@forum.drugs-and-users.org:/etc/ /rsync-live-etc/

rsync-live-root
Code: [Select]
# cat /data/exclusion2
# myswapspace
rsync -zpaAXPv --exclude-from=/data/exclusion2 root@forum.drugs-and-users.org:/root/ /rsync-live-root/

rsync-live-bin
Code: [Select]
rsync -zpaAXPv root@forum.drugs-and-users.org:/bin/ /rsync-live-bin/
rsync-live-git
Code: [Select]
rsync -zpaAXPve ssh root@forum.drugs-and-users.org:/git/ /rsync-live-git/
/data/inclusions
Code: [Select]
index.html
Settings.php

/data/exclusions
Code: [Select]
index.html
Settings.php
config.php
.htaccess
/cache*

/data/exclusions-special
Code: [Select]
smfnew/cache

/data/exclusion2
Code: [Select]
myswapfile
(intranet default webpage to call repair settings etc.) index.html

Code: [Select]
</html>
<head>
<title>"CentOS6 dev/mirror"</title>
</head>
<body>

<h3>This is "index.html" on "192.168.0.180" (CentOS6/ dev/mirror) and all applications are listed below:</h3>

<h3>"http://192.168.0.180" (the site by IP)</h3>
<a href="http://192.168.0.180/"> the site by IP </a>

<h3>"http://dev.mirror" (the site by DNS)</h3>
<a href="http://dev.mirror"> the site by DNS </a>

<h3>"http://192.168.0.180" (repair_settings.php)</h3>
<a href="http://192.168.0.180/repair_settings.php"> repair_settings.php </a>

<h3>"http://192.168.0.180/smfnew" (Vanilla)</h3>
<a href="http://192.168.0.180/smfnew"> SMF Vanilla </a>

<h3>"http://192.168.0.180" (phpMyAdmin)</h3>
<a href="http://192.168.0.180/phpmyadmin"> phpMyAdmin </a>

<h3>"http://192.168.0.180/webmail" (Email/SquirrelMail)</h3>
<a href="http://192.168.0.180/webmail/src/login.php">Webmail/Email</a>

<h3>"http://192.168.0.180" (AWStats)</h3>
<a href="http://dev.mirror/awstats/awstats.pl"> AWStats </a>

<h3>"https://192.168.0.180:10000" (Webmin)</h3>
<a href="https://192.168.0.180:10000/"> Webmin</a>
</br>

</body>
</html>

my repositories:

Code: [Select]
> epel-testing.repo
> epel.repo
> mysql-community-source.repo
> pgdg-94-centos.repo
> gf.repo
> mod-pagespeed.repo
> webtatic-testing.repo
> webtatic-archive.repo
> CentOS-Vault.repo
> CentOS-Media.repo
> CentOS-fasttrack.repo
> CentOS-Debuginfo.repo
> webtatic.repo
> SQUID.repo
> ce:9.1.repo
> remi-php54.repo
> remi.repo
> mysql-community.repo
> iredmail.repo
> remi-safe.repo
> remi.repo.rpmnew
> remi-php72.repo
> remi-php71.repo
> remi-php70.repo
> remi-php54.repo.rpmnew
> nux-dextop.repo
> CentOS-Base.repo-orig
> CentOS-Base.repo~
> CentOS-Base.repo
> linuxtech.repo
> webmin.repo
> filebeat.repo
« Last Edit: November 26, 2018, 07:24:58 AM by aegersz »
The configuration of my Linux VPS (SMF 2.0 with 145 mods & some assorted manual tweaks) can be found here and notes on my mods can be found here (warning: those links will take you to a drug related forum)