Senin, 29 Agustus 2022

FTP-script upload backup

 

Shell Script to Put File in a FTP Server

  • https://www.geeksforgeeks.org/shell-script-to-put-file-in-a-ftp-server/

The File Transfer Protocol also called FTP is used to transfer files from client to server and vice-versa. It mainly uses port 21 for communication.

Here we can simplify the process of uploading files using FTP. But, before we write a script, let’s look at how to get/put files onto an ftp server directly using commands.

Basic FTP Command:

The commands to start an FTP communication and end it is shown below:

ftp server #Prompts for login details and connects to server
#Example: 
ftp 192.168.0.104

bye           #Terminates the ftp connection and exits ftp

Example:

Shell Script to Put File in a FTP Server

After typing “FTP hostname”, you will be asked for a username and password. If login is successful after entering the details, we start in the FTP user’s home directory on the server. Any file you upload now gets uploaded to this directory. If you have to upload a file in some other directory on the server, you first have to change to that directory using the “cd” command. Note that, using the cd command in an FTP prompt only changes directory on the server, i.e. we will still be in the same directory on our local computer.

Commands in FTP help us to navigate the server’s directories, fetch and upload files from and to the server. To get single or multiple files, we can use commands “get” and “mget” respectively. Similarly, to put single or multiple files, we can use commands “put” and “mput” respectively.

Some important ftp commands:

ls                   #Lists files in server
cd dir                #Change directory in server
get file1.c           #Downloads file1.c
put file.txt       #Uploads file.txt
mput *.c file.txt  #Uploads all c files and file.txt

Example of putting a file in server through FTP prompt:

Shell Script to Put File in a FTP Server

Shell Script to put the file in an FTP server:

#!/bin/bash

# The 3 variables below store server and login details
HOST="192.168.0.104"
USER="user1"
PASSWORD="1234"


# $1 is the first argument to the script
# We are using it as upload directory path
# If it is '.', file is uploaded to current directory.
DESTINATION=$1


# Rest of the arguments are a list of files to be uploaded.
# ${@:2} is an array of arguments without first one.
ALL_FILES="${@:2}"


# FTP login and upload is explained in paragraph below
ftp -inv $HOST <<EOF
user $USER $PASSWORD
cd $DESTINATION
mput $ALL_FILES
bye
EOF

Shell Script to Put File in a FTP Server

The above script requires the following data:

  1. Server’s hostname
  2. Server user’s login details
  3. The directory in which to upload files on the server (passed as an argument to the script)
  4. The list of files to be uploaded to the server (passed as an argument to script)

After logging in to the server, we need to enter FTP commands manually, but by using input redirection we can supply the commands directly in the script. “<<” is used for input redirection and “EOF” is used to mark the beginning and end of the FTP input. 

The “mput” command has been used to upload files as mput can upload either a single file or multiple files.

If login is successful and the files given as input to the script are available, all the files should have been put in the server along with a success message displayed for each file.

The options -inv can also be written as -i -n -v and their functions are explained in the below table:

OptionMeaning
-iDisable interactive mode, so that FTP will not ask for confirmation of each file while using mput command etc. We are using this for convenience while uploading or downloading files
-nDisable auto-login. We have to do this, so we can manually log in using “user” command inside the script
-vEnables verbose mode. This helps us to see the server responses after executing each FTP command

To execute the script supply the upload directory and also a list of files:

./script_name.sh  path_to_upload file1 file2 file3

File Uploading Example (Put all .c files and f1.txt in the current directory of server):

Shell Script to Put File in a FTP Server

Kamis, 25 Agustus 2022

POSTGRESQL-barman backup monitoring

 


Barman

Dashboard for postgres-barman exporter https://github.com/ahes/prometheus-barman-exporter

  • Overview
  • Revisions
  • Reviews

You are strongly recommened to try the new dashboard: https://grafana.com/grafana/dashboards/13918

It suports monitoring multiple servers, giving you a summary view.

=============================

Barman exporter for Prometheus

The barman exporter runs barman shell command with experimental JSON output. I am the author of JSON output in Barman so it should work fine until somebody else changes output format which may happen in the future.

By default barman-exporter runs as a service and binds to 127.0.0.1:9780. Metrics are cached and refreshed every hour.

You can run barman-exporter from cron using -f argument to output results to a textfile:

/usr/local/bin/barman-exporter -f /var/lib/prometheus/node_exporter/barman.prom

In such case the node_exporter must point to this path with --collector.textfile.directory option.

Grafana dashboard

You can find basic grafana dashboard in grafana-dashboard.json. It is open for improvements.

Grafana screenshot

Usage

usage: barman-exporter [-h] [-u USER] [-g GROUP] [-m MODE] [-c SECONDS] [-v] [-f TEXTFILE_PATH | -l HOST:PORT | -d]
                       [servers [servers ...]]

Barman exporter

positional arguments:
  servers               Space separated list of servers to check (default: ['all'])

optional arguments:
  -h, --help            show this help message and exit
  -u USER, --user USER  Textfile owner (default: prometheus)
  -g GROUP, --group GROUP
                        Textfile group (default: prometheus)
  -m MODE, --mode MODE  Textfile mode (default: 0644)
  -c SECONDS, --cache-time SECONDS
                        Number of seconds to cache barman output for (default: 3600)
  -v, --version         Show barman exporter version (default: False)
  -f TEXTFILE_PATH, --file TEXTFILE_PATH
                        Save output to textfile (default: None)
  -l HOST:PORT, --listen-address HOST:PORT
                        Address to listen on (default: 127.0.0.1:9780)
  -d, --debug           Print output to stdout (default: False)

Examples:

  • $ /usr/local/bin/barman-exporter postgres-01
  • $ /usr/local/bin/barman-exporter postgres-01 postgres-02
  • $ /usr/local/bin/barman-exporter all
  • $ /usr/local/bin/barman-exporter -l 10.10.10.10:9780 -c 900
  • $ /usr/local/bin/barman-exporter -f /var/lib/prometheus/node_exporter/barman.prom -u prometheus -g prometheus -m 0640 all

Requirements

Python3 and following modules are required to run it:

  • prometheus_client
  • sh

All dependencies will be installed automatically with pip command (see Installation).

Installation

pip3 install barman-exporter

Systemd service file to run barman-exporter as a service

[Unit]
Description=Barman Exporter
After=network-online.target

[Service]
Type=simple
User=barman
Group=barman
ExecStart=/usr/local/bin/barman-exporter -l 10.10.10.10:9780 -c 3600
SyslogIdentifier=barman_exporter
Restart=always

[Install]
WantedBy=multi-user.target

Cron job to run barman-exporter with textfile output

If you don't want to use barman exporter as a service you can run it with -f argument from the cron job. To run it every hour:

0 * * * * /usr/local/bin/barman-exporter -f /var/lib/prometheus/node_exporter/barman.prom

In this mode barman exporter does not require any Prometheus configuration because it uses node-exporter to parse the metrics from a textfile. Remember to use --collector.textfile.directory in node-exporter to define a directory with textfiles.

Prometheus configuration

Please note that barman-exporter is listing all backups which is quite heavy operation to perform and it takes some time. Barman exporter caches its results because execution every 5 seconds would be impossible.

scrape_configs:
  - job_name: barman
    static_configs:
      - targets:
        - 10.10.10.10:9780'

Metrics

  • number=1 label indicates the newest backup
  • barman_backups_size and barman_backup_wal_size show successful backups only. Failed backups will not be listed here.
  • barman_backups_total includes failed backups
  • barman_backups_failedexposes the number of failed backups.
  • barman_last_backup_copy_time shows how long it takes to make a backup
  • barman_up shows all checks from barman check SERVER_NAME command. Output OK is 1.0FAILED is 0.0.
  • barman_metrics_update shows a timestamp when barman metrics has been last updated

With barman_last_backup and barman_first_backup you can easily calculate when the latest backup was completed:

time() - barman_last_backup{instance="$instance", server="$server"}

Raw metrics

# HELP barman_backup_size Size of available backups
# TYPE barman_backup_size gauge
barman_backup_size{number="1",server="postgres-01"} 1.429365116108e+012
barman_backup_size{number="2",server="postgres-01"} 1.429365116108e+012
barman_backup_size{number="3",server="postgres-01"} 1.429365116108e+012
barman_backup_size{number="4",server="postgres-01"} 1.429365116108e+012
barman_backup_size{number="5",server="postgres-01"} 1.429365116108e+012
barman_backup_size{number="6",server="postgres-01"} 1.429365116108e+012
barman_backup_size{number="7",server="postgres-01"} 1.429365116108e+012
barman_backup_size{number="8",server="postgres-01"} 1.429365116108e+012

# HELP barman_backup_wal_size WAL size of available backups
# TYPE barman_backup_wal_size gauge
barman_backup_wal_size{number="1",server="postgres-01"} 1.94347270144e+011
barman_backup_wal_size{number="2",server="postgres-01"} 3.06553290752e+011
barman_backup_wal_size{number="3",server="postgres-01"} 3.05479548928e+011
barman_backup_wal_size{number="4",server="postgres-01"} 4.79318350233e+011
barman_backup_wal_size{number="5",server="postgres-01"} 2.87333312102e+011
barman_backup_wal_size{number="6",server="postgres-01"} 2.73267294208e+011
barman_backup_wal_size{number="7",server="postgres-01"} 3.65501716889e+011
barman_backup_wal_size{number="8",server="postgres-01"} 2.34075717632e+011

# HELP barman_backups_total Total number of backups
# TYPE barman_backups_total gauge
barman_backups_total{server="postgres-01"} 9.0

# HELP barman_backups_failed Number of failed backups
# TYPE barman_backups_failed gauge
barman_backups_failed{server="postgres-01"} 1.0

# HELP barman_last_backup Last successful backup timestamp
# TYPE barman_last_backup gauge
barman_last_backup{server="postgres-01"} 1.562537102e+09

# HELP barman_last_backup_copy_time Last successful backup copy time
# TYPE barman_last_backup_copy_time gauge
barman_last_backup_copy_time{server="postgres-01"} 18706.918297

# HELP barman_first_backup First successful backup timestamp
# TYPE barman_first_backup gauge
barman_first_backup{server="postgres-01"} 1.561154701e+09

# HELP barman_up Barman status checks
# TYPE barman_up gauge
barman_up{check="archiver_errors",server="postgres-01"} 1.0
barman_up{check="backup_maximum_age",server="postgres-01"} 1.0
barman_up{check="compression_settings",server="postgres-01"} 1.0
barman_up{check="directories",server="postgres-01"} 1.0
barman_up{check="failed_backups",server="postgres-01"} 1.0
barman_up{check="is_superuser",server="postgres-01"} 1.0
barman_up{check="minimum_redundancy_requirements",server="postgres-01"} 1.0
barman_up{check="pg_basebackup",server="postgres-01"} 1.0
barman_up{check="pg_basebackup_compatible",server="postgres-01"} 1.0
barman_up{check="pg_basebackup_supports_tablespaces_mapping",server="postgres-01"} 1.0
barman_up{check="pg_receivexlog",server="postgres-01"} 1.0
barman_up{check="pg_receivexlog_compatible",server="postgres-01"} 1.0
barman_up{check="postgresql",server="postgres-01"} 1.0
barman_up{check="postgresql_streaming",server="postgres-01"} 1.0
barman_up{check="receive_wal_running",server="postgres-01"} 1.0
barman_up{check="replication_slot",server="postgres-01"} 1.0
barman_up{check="retention_policy_settings",server="postgres-01"} 1.0
barman_up{check="systemid_coherence",server="postgres-01"} 1.0
barman_up{check="wal_level",server="postgres-01"} 1.0

# HELP barman_metrics_update Barman metrics update timestamp
# TYPE barman_metrics_update gauge
barman_metrics_update{server="autouncle"} 1.580485601e+09

Development

Upload to PyPi:

source venv/bin/activate
rm -f dist/*
python3 setup.py sdist
twine upload dist/*

POSTGRSQL-backup central barman

 

Install and Configure BARMAN for PostgreSQL


BARMAN (Backup and Recovery Manager) is an open-source administration tool for backup and disaster recovery of PostgreSQL servers. It allows performing remote backups of multiple servers. Following are barman features

      1) Open Source Tool.
      2) Remote backup and restore of multiple Servers.
      3) Backup catalogs.
      4) Incremental backup is possible.
      5) You can define retention policies.
      6) Archiving and compression of Wal files and backup files.
      7) Support both rsync or postgresql protocols.

Sr NoHostnameIPRole
1test-machine01192.168.114.177Barman Server (Backup Server)
2test-machine02192.168.114.176PostgreSQL Database Server


Follow below steps to install and configure barman


Step 1. Verify PostgresQL repository: Use OS command yum repolist to confirm postgresql repository exist in Backup ServerClick here to configure postgresql repository if it is not configured.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
[root@test-machine01 ~]# yum repolist
Loaded plugins: langpacks, ulninfo
repo id                                                              repo name                                                                                                          status
!mysql-connectors-community/x86_64                                   MySQL Connectors Community                                                                                            203
!mysql-tools-community/x86_64                                        MySQL Tools Community                                                                                                 129
!mysql80-community/x86_64                                            MySQL 8.0 Community Server                                                                                            265
!ol7_UEKR5/x86_64                                                    Latest Unbreakable Enterprise Kernel Release 5 for Oracle Linux 7Server (x86_64)                                      404
!ol7_UEKR6/x86_64                                                    Latest Unbreakable Enterprise Kernel Release 6 for Oracle Linux 7Server (x86_64)                                      323
!ol7_latest/x86_64                                                   Oracle Linux 7Server Latest (x86_64)                                                                               22,766
!pgdg-common/7Server/x86_64                                          PostgreSQL common RPMs for RHEL/CentOS 7Server - x86_64                                                               303
!pgdg10/7Server/x86_64                                               PostgreSQL 10 for RHEL/CentOS 7Server - x86_64                                                                        918
!pgdg11/7Server/x86_64                                               PostgreSQL 11 for RHEL/CentOS 7Server - x86_64                                                                      1,006
!pgdg12/7Server/x86_64                                               PostgreSQL 12 for RHEL/CentOS 7Server - x86_64                                                                        572
!pgdg13/7Server/x86_64                                               PostgreSQL 13 for RHEL/CentOS 7Server - x86_64                                                                        298
!pgdg95/7Server/x86_64                                               PostgreSQL 9.5 for RHEL/CentOS 7Server - x86_64                                                                       763
!pgdg96/7Server/x86_64                                               PostgreSQL 9.6 for RHEL/CentOS 7Server - x86_64                                                                       889
repolist: 28,839
[root@test-machine01 ~]#


Step 2. Install barman: Use OS command yum install barman to install barman package in Backup Server.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
[root@test-machine01 tmp]# yum install barman
Loaded plugins: langpacks, ulninfo
Resolving Dependencies
--> Running transaction check
---> Package barman.noarch 0:2.12-1.rhel7 will be installed
--> Processing Dependency: python-barman = 2.12 for package: barman-2.12-1.rhel7.noarch
--> Running transaction check
---> Package python-barman.noarch 0:2.12-1.rhel7 will be installed
--> Processing Dependency: python-psycopg2 >= 2.4.2 for package: python-barman-2.12-1.rhel7.noarch
--> Running transaction check
---> Package python2-psycopg2.x86_64 0:2.8.6-1.rhel7 will be installed
--> Finished Dependency Resolution
 
Dependencies Resolved
 
==============================================================================================================================================================================================
 Package                                           Arch                                    Version                                         Repository                                    Size
==============================================================================================================================================================================================
Installing:
 barman                                            noarch                                  2.12-1.rhel7                                    pgdg-common                                   36 k
Installing for dependencies:
 python-barman                                     noarch                                  2.12-1.rhel7                                    pgdg-common                                  356 k
 python2-psycopg2                                  x86_64                                  2.8.6-1.rhel7                                   pgdg-common                                  171 k
 
Transaction Summary
==============================================================================================================================================================================================
Install  1 Package (+2 Dependent packages)
 
Total download size: 563 k
Installed size: 2.5 M
Is this ok [y/d/N]: y
Downloading packages:
(1/3): barman-2.12-1.rhel7.noarch.rpm                                                                                                                                  |  36 kB  00:00:01
(2/3): python2-psycopg2-2.8.6-1.rhel7.x86_64.rpm                                                                                                                       | 171 kB  00:00:01
(3/3): python-barman-2.12-1.rhel7.noarch.rpm                                                                                                                           | 356 kB  00:00:03
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Total                                                                                                                                                         176 kB/s | 563 kB  00:00:03
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
Warning: RPMDB altered outside of yum.
  Installing : python2-psycopg2-2.8.6-1.rhel7.x86_64                                                                                                                                      1/3
  Installing : python-barman-2.12-1.rhel7.noarch                                                                                                                                          2/3
  Installing : barman-2.12-1.rhel7.noarch                                                                                                                                                 3/3
  Verifying  : python2-psycopg2-2.8.6-1.rhel7.x86_64                                                                                                                                      1/3
  Verifying  : barman-2.12-1.rhel7.noarch                                                                                                                                                 2/3
  Verifying  : python-barman-2.12-1.rhel7.noarch                                                                                                                                          3/3
 
Installed:
  barman.noarch 0:2.12-1.rhel7
 
Dependency Installed:
  python-barman.noarch 0:2.12-1.rhel7                                                         python2-psycopg2.x86_64 0:2.8.6-1.rhel7
 
Complete!
[root@test-machine01 tmp]#


Step 3. In case of Error: In case if you encounter errors Error: Package: python-barman-2.12-1.rhel7.noarch (pgdg-common) Requires: python-argcomplete & Error: Package: python-barman-2.12-1.rhel7.noarch (pgdg-common) Requires: python-argh >= 0.21.2 in yum install barman command. Download rpm python2-argh-0.26.1-5.el7.noarch.rpm & python2-argcomplete-1.7.0-4.el7.noarch.rpm from below links and transfer these rpm to Backup Server and install the rpm and then try to install barman package.

https://download-ib01.fedoraproject.org/pub/epel/7/x86_64/Packages/p/python2-argcomplete-1.7.0-4.el7.noarch.rpm
https://download-ib01.fedoraproject.org/pub/epel/7/aarch64/Packages/p/python2-argh-0.26.1-5.el7.noarch.rpm

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
[root@test-machine01 u01]# yum install barman
Loaded plugins: langpacks, ulninfo
Repository ol7_UEKR5 is listed more than once in the configuration
Repository ol7_UEKR4 is listed more than once in the configuration
Repository ol7_UEKR3 is listed more than once in the configuration
---> Package python-backports.x86_64 0:1.0-8.el7 will be installed
---> Package python-barman.noarch 0:2.12-1.rhel7 will be installed
--> Processing Dependency: python-argh >= 0.21.2 for package: python-barman-2.12-1.rhel7.noarch
--> Processing Dependency: python-argcomplete for package: python-barman-2.12-1.rhel7.noarch
---> Package python-ipaddress.noarch 0:1.0.16-2.el7 will be installed
--> Finished Dependency Resolution
Error: Package: python-barman-2.12-1.rhel7.noarch (pgdg-common)
           Requires: python-argcomplete
Error: Package: python-barman-2.12-1.rhel7.noarch (pgdg-common)
           Requires: python-argh >= 0.21.2
 You could try using --skip-broken to work around the problem
 You could try running: rpm -Va --nofiles --nodigest
[root@test-machine01 u01]#
 
root@test-machine01 u01]# cd /tmp
[root@test-machine01 tmp]#
[root@test-machine01 tmp]# ls -ltr
total 156
-rw-r--r--. 1 root root 55120 Jun 22 09:57 python2-argh-0.26.1-5.el7.noarch.rpm
-rw-r--r--. 1 root root 50402 Jun 22  2021 python2-argcomplete-1.7.0-4.el7.noarch.rpm
[root@test-machine01 u01]#
 
[root@test-machine01 tmp]# rpm -ivh python2-argh-0.26.1-5.el7.noarch.rpm
warning: python2-argh-0.26.1-5.el7.noarch.rpm: Header V3 RSA/SHA256 Signature, key ID 352c64e5: NOKEY
Preparing...                          ################################# [100%]
Updating / installing...
   1:python2-argh-0.26.1-5.el7        ################################# [100%]
[root@test-machine01 tmp]#
 
[root@test-machine01 tmp]# rpm -ivh python2-argcomplete-1.7.0-4.el7.noarch.rpm
warning: python2-argcomplete-1.7.0-4.el7.noarch.rpm: Header V3 RSA/SHA256 Signature, key ID 352c64e5: NOKEY
Preparing...                          ################################# [100%]
Updating / installing...
   1:python2-argcomplete-1.7.0-4.el7  ################################# [100%]
[root@test-machine01 tmp]#
 
 
 
[root@test-machine01 tmp]# yum install barman
Loaded plugins: langpacks, ulninfo
Resolving Dependencies
--> Running transaction check
---> Package barman.noarch 0:2.12-1.rhel7 will be installed
--> Processing Dependency: python-barman = 2.12 for package: barman-2.12-1.rhel7.noarch
--> Running transaction check
---> Package python-barman.noarch 0:2.12-1.rhel7 will be installed
--> Processing Dependency: python-psycopg2 >= 2.4.2 for package: python-barman-2.12-1.rhel7.noarch
--> Running transaction check
---> Package python2-psycopg2.x86_64 0:2.8.6-1.rhel7 will be installed
--> Finished Dependency Resolution
 
Dependencies Resolved
 
==============================================================================================================================================================================================
 Package                                           Arch                                    Version                                         Repository                                    Size
==============================================================================================================================================================================================
Installing:
 barman                                            noarch                                  2.12-1.rhel7                                    pgdg-common                                   36 k
Installing for dependencies:
 python-barman                                     noarch                                  2.12-1.rhel7                                    pgdg-common                                  356 k
 python2-psycopg2                                  x86_64                                  2.8.6-1.rhel7                                   pgdg-common                                  171 k
 
Transaction Summary
==============================================================================================================================================================================================
Install  1 Package (+2 Dependent packages)
 
Total download size: 563 k
Installed size: 2.5 M
Is this ok [y/d/N]: y
Downloading packages:
(1/3): barman-2.12-1.rhel7.noarch.rpm                                                                                                                                  |  36 kB  00:00:01
(2/3): python2-psycopg2-2.8.6-1.rhel7.x86_64.rpm                                                                                                                       | 171 kB  00:00:01
(3/3): python-barman-2.12-1.rhel7.noarch.rpm                                                                                                                           | 356 kB  00:00:03
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Total                                                                                                                                                         176 kB/s | 563 kB  00:00:03
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
Warning: RPMDB altered outside of yum.
  Installing : python2-psycopg2-2.8.6-1.rhel7.x86_64                                                                                                                                      1/3
  Installing : python-barman-2.12-1.rhel7.noarch                                                                                                                                          2/3
  Installing : barman-2.12-1.rhel7.noarch                                                                                                                                                 3/3
  Verifying  : python2-psycopg2-2.8.6-1.rhel7.x86_64                                                                                                                                      1/3
  Verifying  : barman-2.12-1.rhel7.noarch                                                                                                                                                 2/3
  Verifying  : python-barman-2.12-1.rhel7.noarch                                                                                                                                          3/3
 
Installed:
  barman.noarch 0:2.12-1.rhel7
 
Dependency Installed:
  python-barman.noarch 0:2.12-1.rhel7                                                         python2-psycopg2.x86_64 0:2.8.6-1.rhel7
 
Complete!
[root@test-machine01 tmp]#


Step 4. Verify barman package installation: yum install barman will create barman account in Server. Use command barman –version to verify install barman version in Backup Server.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
[root@test-machine01 tmp]# id barman
uid=994(barman) gid=990(barman) groups=990(barman)
[root@test-machine01 tmp]#
[root@test-machine01 tmp]# su - barman
-bash-4.2$
-bash-4.2$ whoami
barman
-bash-4.2$
 
-bash-4.2$ barman --version
2.12
 
Barman by 2ndQuadrant (www.2ndQuadrant.com)
-bash-4.2$


Step 5. Set Password for OS Account barman: Use OS command passwd barman to reset barman OS account in Backup Server.

1
2
3
4
5
6
[root@test-machine01 tmp]# passwd barman
Changing password for user barman.
New password:
Retype new password:
passwd: all authentication tokens updated successfully.
[root@test-machine01 tmp]#


Step 6. Set Password for OS Account postgres: Use OS command passwd postgres to reset postgres OS account in PostgreSQL Database Server.

1
2
3
4
5
6
7
[root@test-machine02 ~]# passwd postgres
Changing password for user postgres.
New password:
BAD PASSWORD: The password contains the user name in some form
Retype new password:
passwd: all authentication tokens updated successfully.
[root@test-machine02 ~]#


Step 7. Create barman role: Create role barman in PostgreSQL Database Server. This role will use in barman server configuration file.

1
2
3
4
5
6
7
8
9
[root@test-machine02 ~]# su - postgres
Last login: Tue Jun 22 15:06:21 +03 2021 on pts/0
-bash-4.2$ psql
psql (13.2)
Type "help" for help.
postgres=#
postgres=# create user barman superuser  password 'barman';
CREATE ROLE
postgres=#


Step 8. Configure Passwordless Authentication in Barman Server: Use OS command ssh-keygen to generate public and private keys. Once keys are generated use OS command ssh-copy-id to transfer public key to PostgreSQL Database Server.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
[root@test-machine01 ~]# su - barman
Last login: Tue Jun 22 10:02:53 +03 2021 on pts/0
-bash-4.2$
-bash-4.2$ ssh-keygen -t rsa
Generating public/private rsa key pair.
Enter file in which to save the key (/var/lib/barman/.ssh/id_rsa):
Created directory '/var/lib/barman/.ssh'.
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /var/lib/barman/.ssh/id_rsa.
Your public key has been saved in /var/lib/barman/.ssh/id_rsa.pub.
The key fingerprint is:
SHA256:fg8qzAkNeneq4qKoYRNJCTNP5+S7z4RQWXb889KpETA barman@test-machine01
The key's randomart image is:
+---[RSA 2048]----+
|+ . o o..        |
|.=.= + .E        |
| o. =    +       |
|. ....    +      |
| o...o  S  = .   |
|  o.ooo.. o +    |
|.o .o=.+. o+     |
|+.o  +*  o.o     |
|*o....o..   .    |
+----[SHA256]-----+
-bash-4.2$
-bash-4.2$ls -ltr .ssh
total 8
-rw-r--r--. 1 barman barman  403 Jun 22 10:12 id_rsa.pub
-rw-------. 1 barman barman 1675 Jun 22 10:12 id_rsa
-bash-4.2$
 
-bash-4.2$ ssh-copy-id -i ~/.ssh/id_rsa.pub postgres@test-machine02
/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/var/lib/barman/.ssh/id_rsa.pub"
The authenticity of host 'test-machine02 (192.168.114.176)' can't be established.
ECDSA key fingerprint is SHA256:NbJdl0Pi8lmjbKHbz7q0GgYAy++evY3rNH7gCupnx+c.
ECDSA key fingerprint is MD5:39:33:1f:a7:bb:09:ac:81:cc:ff:cc:18:46:ca:76:c1.
Are you sure you want to continue connecting (yes/no)? yes
/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
postgres@test-machine02's password:
 
Number of key(s) added: 1
 
Now try logging into the machine, with:   "ssh 'postgres@test-machine02'"
and check to make sure that only the key(s) you wanted were added.
 
-bash-4.2$
-bash-4.2$ ssh 'postgres@test-machine02'
Last login: Tue Jun 22 10:30:29 2021
-bash-4.2$
-bash-4.2$ exit
logout
Connection to test-machine02 closed.
-bash-4.2$


Step 9. Configure Passwordless Authentication in PostgreSQL Database Server: Use OS command ssh-keygen to generate public and private keys. Once keys are generated use OS command ssh-copy-id to transfer public key to Barman Server.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
[root@test-machine02 ~]# su - postgres
Last login: Wed Jun 16 17:44:29 +03 2021 on pts/2
-bash-4.2$
-bash-4.2$
-bash-4.2$ ssh-keygen -t rsa
Generating public/private rsa key pair.
Enter file in which to save the key (/var/lib/pgsql/.ssh/id_rsa):
Created directory '/var/lib/pgsql/.ssh'.
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /var/lib/pgsql/.ssh/id_rsa.
Your public key has been saved in /var/lib/pgsql/.ssh/id_rsa.pub.
The key fingerprint is:
SHA256:jiouqVEgyZz+dmsHM+H0yzn4GgdhYnqBkbJ3Es/ralw postgres@test-machine02
The key's randomart image is:
+---[RSA 2048]----+
| ..              |
|+o=              |
|=* B o           |
|+.= Bo.          |
| +.+ooo S        |
| .o E=.+         |
|.o = oB.+        |
|o.= o++*         |
|o+oooo+..        |
+----[SHA256]-----+
-bash-4.2$
 
-bash-4.2$ ls -ltr .ssh
total 8
-rw-r--r--. 1 postgres postgres  405 Jun 22 10:12 id_rsa.pub
-rw-------. 1 postgres postgres 1679 Jun 22 10:12 id_rsa
-bash-4.2$
 
-bash-4.2$ ssh-copy-id -i ~/.ssh/id_rsa.pub barman@test-machine01
/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/var/lib/pgsql/.ssh/id_rsa.pub"
/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
barman@test-machine01's password:
 
Number of key(s) added: 1
 
Now try logging into the machine, with:   "ssh 'barman@test-machine01'"
and check to make sure that only the key(s) you wanted were added.
 
-bash-4.2$
-bash-4.2$ ssh 'barman@test-machine01'
Last login: Tue Jun 22 10:32:10 2021
-bash-4.2$
-bash-4.2$ exit
logout
Connection to test-machine01 closed.
-bash-4.2$


Step 10. Configure Global Configuration File in Barman Server: The main configuration file (set to /etc/barman.conf by default) contains general options such as main directory, system user, log file, and so on. Edit /etc/barman.conf to set global parameters. Set the below parameters and save the file.

1
2
3
4
5
6
7
8
9
10
11
[root@test-machine01 ~]#
[root@test-machine01 ~]# ls -ltr /etc/barman.conf
-rw-r--r--. 1 root root 3195 Nov  4  2020 /etc/barman.conf
[root@test-machine01 ~]# cp /etc/barman.conf /etc/barman_bkp.conf
[root@test-machine01 ~]# vi /etc/barman.conf
compression = gzip
immediate_checkpoint = true
basebackup_retry_times = 3
reuse_backup = link
:wq!
[root@test-machine01 ~]#


Step 11. Configure Server configuration files in Barman Server: Server configuration files, one for each server to be backed up by Barman, is located in the /etc/barman.d directory and must have a .conf suffix. Please note user=barman is the same role we created in Step 6. Set the below parameters and save the file.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
root@test-machine01 ~]#
[root@test-machine01 ~]# cd /etc/barman.d
[root@test-machine01 barman.d]#
[root@test-machine01 barman.d]# ls -ltr
total 12
-rw-r--r--. 1 root root 1492 Nov  4  2020 streaming-server.conf-template
-rw-r--r--. 1 root root 1565 Nov  4  2020 ssh-server.conf-template
-rw-r--r--. 1 root root  947 Nov  4  2020 passive-server.conf-template
[root@test-machine01 barman.d]# vi test-machine02.conf
[test-machine02]
description =  "test-machine02 backup config"
ssh_command = ssh postgres@test-machine02
conninfo = host=test-machine02 user=barman port=5432 dbname=postgres password=barman
backup_options = concurrent_backup
backup_method = rsync
archiver = on
:wq!
 
[root@test-machine01 barman.d]# ls -ltr
total 16
-rw-r--r--. 1 root root 1492 Nov  4  2020 streaming-server.conf-template
-rw-r--r--. 1 root root 1565 Nov  4  2020 ssh-server.conf-template
-rw-r--r--. 1 root root  947 Nov  4  2020 passive-server.conf-template
-rw-r--r--. 1 root root  226 Jun 22 11:50 test-machine02.conf
[root@test-machine01 barman.d]#


Step 12. Verify Server setup in Barman Server: Use command barman list-server to Server list configured in barman. Command barman show-server test-machine02 to get server configuration setting. And command barman check test-machine02 to validate the setup. Please note we are getting an error “WAL archive: FAILED (please make sure WAL shipping is setup)“.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
[root@test-machine01 barman.d]# su - barman
Last login: Tue Jun 22 11:59:11 +03 2021 on pts/0
-bash-4.2$
 
-bash-4.2$ barman list-server
test-machine02 - test-machine02 backup config
-bash-4.2$
 
-bash-4.2$ barman show-server test-machine02
Server test-machine02:
        active: True
        archive_command: None
        archive_mode: None
        archiver: True
        archiver_batch_size: 0
 
-bash-4.2$  barman check test-machine02
Server test-machine02:
        WAL archive: FAILED (please make sure WAL shipping is setup)
        PostgreSQL: OK
        superuser or standard user with backup privileges: OK
        wal_level: OK
        directories: OK
        retention policy settings: OK
        backup maximum age: OK (no last_backup_maximum_age provided)
        compression settings: OK
        failed backups: OK (there are 0 failed backups)
        minimum redundancy requirements: OK (have 0 backups, expected at least 0)
        ssh: OK (PostgreSQL server)
        systemid coherence: OK (no system Id stored on disk)
        archive_mode: OK
        archive_command: OK
        continuous archiving: OK
        archiver errors: OK
-bash-4.2$


Step 13. Configure WAL Shipping in PostgreSql Database Server: Edit postgresql.conf and modify archive_command to copy WAL Files in barman server. As archive_command is a static parameter postgresql service needs to be restart.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
[root@test-machine02 ~]# cd /var/lib/pgsql/13/data/
[root@test-machine02 data]# vi postgresql.conf
archive_command = 'test ! -f barman@test-machine01:/var/lib/barman/test-machine02/incoming/%f && rsync -a %p barman@test-machine01:/var/lib/barman/test-machine02/incoming/%f'
:wq!
 
[root@test-machine02 system]# systemctl restart postgresql-13
[root@test-machine02 system]#
 
[root@test-machine02 ~]#
[root@test-machine02 ~]# su - postgres
Last login: Tue Jun 22 10:34:26 +03 2021 from test-machine01 on pts/3
-bash-4.2$ psql
psql (13.2)
Type "help" for help.
 
postgres=# show archive_command;
                                                                      archive_command
------------------------------------------------------------------------------------------------------------------------------------------------------------
 test ! -f barman@test-machine01:/var/lib/barman/test-machine02/incoming/%f && rsync -a %p barman@test-machine01:/var/lib/barman/test-machine02/incoming/%f
(1 row)
 
postgres=#


Step 14. Force Switch WAL in Barman Server: Use barman command switch-xlog –force to perform force switch of WAL File. And run barman check test-machine02 again to verify settings. Now it should be all OK.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
[root@test-machine01 barman.d]# su - barman
Last login: Tue Jun 22 11:59:11 +03 2021 on pts/0
-bash-4.2$
 
-bash-4.2$ barman switch-xlog --force --archive test-machine02
The WAL file 00000001000000000000002E has been closed on server 'test-machine02'
Waiting for the WAL file 00000001000000000000002E from server 'test-machine02' (max: 30 seconds)
Processing xlog segments from file archival for test-machine02
        00000001000000000000002E
-bash-4.2$
-bash-4.2$ barman check test-machine02
Server test-machine02:
        PostgreSQL: OK
        superuser or standard user with backup privileges: OK
        wal_level: OK
        directories: OK
        retention policy settings: OK
        backup maximum age: OK (no last_backup_maximum_age provided)
        compression settings: OK
        failed backups: OK (there are 0 failed backups)
        minimum redundancy requirements: OK (have 0 backups, expected at least 0)
        ssh: OK (PostgreSQL server)
        systemid coherence: OK (no system Id stored on disk)
        archive_mode: OK
        archive_command: OK
        continuous archiving: OK
        archiver errors: OK
-bash-4.2$


Reference: http://docs.pgbarman.org/release/2.12/#installation


This document is just for learning purpose and always validate in the LAB environment first before applying in the LIVE environment.