Author Archive

Automating Remote Backups: A Robust Rsync Retention & Pruning Script

Managing remote backups is a critical task for any sysadmin or developer, but the real challenge isn’t just moving the data—it’s managing the storage. Without a proper retention policy, remote servers can quickly become cluttered with outdated archives, leading to storage overages and manual cleanup headaches.

solve the “sync and prune” challenge, we have developed a specialized bash script designed to handle incremental backups via rsync while maintaining a strict retention policy on the remote target.

At Bizmate Solutions Ltd, we prioritise automation and efficiency. To solve the “sync and prune” challenge, we have developed a specialised bash script designed to handle incremental backups via rsync while maintaining a strict retention policy on the remote target.

The Problem: The Infinite Growth of Backups

Standard rsync commands are excellent for mirroring data, but they don’t inherently manage historical versions or clean up old files on a remote destination without risk. Manual pruning is prone to human error and often gets overlooked until a “Disk Full” alert triggers.

The Solution: Sync & Prune

Our latest addition to the bash-essentials library automates this entire life-cycle. The script ensures that:

1. Data is synchronized securely via SSH.

2. Incremental logic is applied to save bandwidth and time.

3. Automatic Pruning kicks in after the sync, keeping only a specified number of the most recent backups.

Key Features

• Configurable Retention: Easily set how many backup rotations you want to keep.

• Secure Execution: Leverages SSH for encrypted data transfer.

• Detailed Logging: Clear output for monitoring and troubleshooting.

• Dry-Run Support: Test your pruning logic before committing to deletions.

Getting Started

The script is open-source and available on our GitHub repository. You can integrate it into your CRON jobs to achieve a “set and forget” backup strategy.

View the documentation and script here: Bash Essentials: Rsync Remote Retention

Efficient infrastructure starts with smart automation. If your team needs assistance optimising DevOps workflows or data management strategies, visit us at www.bizmate.biz.

ubuntu kernel updates and unstable nvidia drivers

Notice this page is an informal log of some of the debugging and troubleshooting for Nvidia drivers and Kernel updates problems. Nvidia provides Linux/Ubuntu drivers out of the box. Installing them is really easy such as running

`sudo ubuntu-drivers install` .

Quite often though these drivers might not be compatible or still have problems so here we go with some troubleshooting.

Ubuntu drivers not fully installed.

Sometimes despite ubuntu-drivers install suggesting the drivers are indeed already installed we might be in an unstable situation where a new kernel has been installed but when drivers are updated the kernel relative modules are not. As such we could run

`sudo dkms autoinstall && reboot`

Check if driver is indeed installed

Running nvidia-smi gives you an output of the currently installed drivers. If this application is not installed then the proprietary drivers are missing

$ nvidia-smi
Thu Dec 7 11:28:46 2023 
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.129.03 Driver Version: 535.129.03 CUDA Version: 12.2 |
|-----------------------------------------+----------------------+----------------------+
| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+======================+======================|
| 0 NVIDIA GeForce RTX 2070 Off | 00000000:09:00.0 On | N/A |
| 0% 49C P8 22W / 175W | 1142MiB / 8192MiB | 1% Default |
| | | N/A |
+-----------------------------------------+----------------------+----------------------+

+---------------------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=======================================================================================|
| 0 N/A N/A 6270 G /usr/lib/xorg/Xorg 497MiB |
| 0 N/A N/A 6433 G /usr/bin/gnome-shell 105MiB |
| 0 N/A N/A 7347 G /usr/bin/nextcloud 1MiB |
| 0 N/A N/A 7519 G cairo-dock 5MiB |
| 0 N/A N/A 7691 C+G ...83750398,4784756152597305294,262144 276MiB |
| 0 N/A N/A 26615 G ...zmate/jcef_26128.log --shared-files 13MiB |
| 0 N/A N/A 28209 G ...,WinRetrieveSuggestionsOnlyOnDemand 86MiB |
| 0 N/A N/A 40478 G /usr/lib/thunderbird/thunderbird 150MiB |
+---------------------------------------------------------------------------------------+

Troubleshooting errors by checking system messages

Very often boot up or start up errors are also recorded and they could help explaining why there is an error or conflict causing your Nvidia card from not working correctly, for instance displaying/detecting only one of the two monitors plugged in with your card. Journalctl is a great tool to check these errors. To debug from this messages buffer you can

journalctl -kb | less

In many cases you can see errors like

[   5.004707] nvidia-gpu 0000:05:00.3: i2c timeout error e0000000

You can then search for an error. Quite often some errors are indeed conflicts with other boot/start up processes. As such these modules can be blacklisted by adding modprobe specific entries such as blacklisting the ic2_nvidia_gpu if it errors

echo "blacklist i2c_nvidia_gpu" > /etc/modprobe.d/blacklist_i2c-nvidia-gpu.conf

Other troubleshooting commands

• Hardware lister – sudo lshw
• PCI devices listing – sudo lspci
• DMI Table decoder – sudo dmidecode -t bios
• RANDR cli tool – xrandr

Change Kernels

In some cases you might have an old or new kernel and the nvidia driver might be not fully compatible with this kernel. You can use a tooling called mainline, that installed on linux/ubuntu allows to install and set as the main kernel another one other than the one currently installed.

Checking and rolling back migrations with doctrine.

This is a quick note on how to work with doctrine migrations

When a change in the mapping is done you can run

I have no name!@7efd2339724b:/var/www/html$ bin/console doctrine:schema:validate

Mapping
-------


[OK] The mapping files are correct. 


Database
--------


[ERROR] The database schema is not in sync with the current mapping file.

This will show if the mapping files are correct and if indeed the mapping matches the current schema status.

If the status is not in sync then you might need to update, run or generate a migration.

To generate a migration run…

I have no name!@7efd2339724b:/var/www/html$ bin/console doctrine:migrations:diff 
Generated new migration class to "/var/www/html/src/Infrastructure/Persistence/Doctrine/Migrations/Version20231127094706.php"

This new migration will have the new definitions for the schema to upgrate the state to the current state. It is generally a good practice to review it.

However if you want to rollback and run a specific migration this can be done by first checking the migrations status …

I have no name!@7efd2339724b:/var/www/html$ bin/console doctrine:migrations:status
+----------------------+------------------------------------------------------------------------------+--------------------------------------------------------------------------------------------------------+
| Configuration |
+----------------------+------------------------------------------------------------------------------+--------------------------------------------------------------------------------------------------------+
| Storage | Type | Doctrine\Migrations\Metadata\Storage\TableMetadataStorageConfiguration |
| | Table Name | doctrine_migration_versions |
| | Column Name | version |
|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| Database | Driver | Symfony\Bridge\Doctrine\Middleware\Debug\Driver |
| | Name | devdb |
|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| Versions | Previous | 0 |
| | Current | MyApp\YelpBundle\Infrastructure\Persistence\Doctrine\Migrations\Version20230605164531 |
| | Next | MyRApp\ReviewsConfigBundle\Infrastructure\Persistence\Doctrine\Migrations\Version20230118085703 |
| | Latest | MyApp\YelpBundle\Infrastructure\Persistence\Doctrine\Migrations\Version20230605164531 |
|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| Migrations | Executed | 1 |
| | Executed Unavailable | 0 |
| | Available | 2 |
| | New | 1 |
|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| Migration Namespaces | MyApp\ReviewsConfigBundle\Infrastructure\Persistence\Doctrine\Migrations | /var/www/html/src/Infrastructure/Persistence/Doctrine/Migrations |
| | MyApp\YelpBundle\Infrastructure\Persistence\Doctrine\Migrations | /var/www/html/vendor/bizmate/myreviews-yelp-reviews/src/Infrastructure/Persistence/Doctrine/Migrations |

Then you can rollback the migrations by notice in the command line you will need to escape the backward slashes or the migration will not be identified.

I have no name!@7efd2339724b:/var/www/html$ bin/console doctrine:migrations:execute MyApp\\ReviewsConfigBundle\\Infrastructure\\Persistence\\Doctrine\\Migrations\\Version20230118085703 --down

WARNING! You are about to execute a migration in database "devdb" that could result in schema changes and data loss. Are you sure you wish to continue? (yes/no) [yes]:
> yes

[notice] Executing MyApp\ReviewsConfigBundle\Infrastructure\Persistence\Doctrine\Migrations\Version20230118085703 down
[notice] finished in 46.2ms, used 18M memory, 1 migrations executed, 23 sql queries

[OK] Successfully migrated version(s): 
MyApp\ReviewsConfigBundle\Infrastructure\Persistence\Doctrine\Migrations\Version20230118085703: [DOWN]

NOTICE if multiple bundles are registered, to keep track of the other bundles migrations status.

Give meaningful names to your Natwest bank statements using bash and phptotext

Why are certain banks so bad when it comes to naming your e-statements from your accounts?

Why when you download a statement from your account you get a PDF names like “b5862f22-c86f-4195-a756-8ea2e009da85.pdf“.

Would it help if the file had a more meaningful name such as “MYACCOUNT-NAME_ACCOUNT-NUMBER_YYYYMMDD.pdf” where you know from the name exactly what account this belongs to and a date that the file is related too?

Try a bash script to rename all files after you download them

Well Natwest in the UK does not care about fixing this small but simple problem and so here comes a small script written in bash and using pdftotxt to find the end date of the statement and give a meaningful name to your Natwest statements.

Please note that the script is not given with any sort of warranty so please the script description and the license file in the repository.

By using the script you accept the disclaimer.

I hope this script helps.

linux bash file rename with dates formatted as DD-MM-YYYY to YYYYMMDD

I am often, as I assume several other people, served with files from online services that have rather strange or incomplete formats and information on their name.

Especially for bank accounts I like an element of semantic description, such as “CreditCard”, “{bankName}_statement”, etc. followed by a well formatted date such as YYYYMMDD, standing for year, month and day such as 20230228.

This date formatting helps because despite the file creation time it still allows ordering the files by name and having the year and month before the day will show the files ordered according to this part of the name. For example a file named “CreditCard_30032021.pdf” will show after a file called “CreditCard_29032022.pdf” if ordering by name, even if older.

See example of a run here:

As a solution for this specific problem i just created a bash script, supposed to work with bash 4.4 and after, so it should work in any modern linux distribution.

The script is available on github but in show it is just a function that splits the initial DD-MM-YYYY format into day, month and year. It then removes the old date format from the file name and replaces it with YYYYMMDD.

Please give it a try first without confirming any changes/renaming and if you have any questions or comments feel free to contact us.

AWS S3 Cli quick reference

Dreamhosts these days forces use of DreamObjects. This is just a custom wrapper of S3 and so it can be used in a very similar way as you would use the original. Simplified steps

Set up credentials

$ export AWS_ACCESS_KEY_ID=AKIAIOSFODNN7EXAMPLE
$ export AWS_SECRET_ACCESS_KEY=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
$ export AWS_DEFAULT_REGION=us-west-2

List content in the bucket (get the bucket from the Dreamhost panel if needs DreamObjects)

aws –endpoint-url https://objects-us-east-1.dream.io s3 ls

Download whole directory

aws –endpoint-url https://objects-us-east-1.dream.io s3 cp s3://trippinrecords-20220427-dh-data-backup/db_backup . –recursive

Delete directory

aws –endpoint-url https://objects-us-east-1.dream.io s3 rm s3://trippinrecords-20220427-dh-data-backup/ –recursive