• info@bizmate.biz

IT Articles

Clear up space/delete file being held up by a process in linux

Recently I had to troubleshoot a disk space problem, hard disk full on our client server, caused by a docker process incorrectly writing files in its local file system instead of forwarding them to stdout.

# df on / it shows 347G used and only 44 G available
df -h
Filesystem      Size  Used Avail Use% Mounted on
udev             32G     0   32G   0% /dev
tmpfs           6.3G  640M  5.7G  10% /run
/dev/md2        438G  372G   44G  90% /
tmpfs            32G  3.4M   32G   1% /dev/shm
tmpfs           5.0M     0  5.0M   0% /run/lock
tmpfs            32G     0   32G   0% /sys/fs/cgroup
/dev/md1        488M  486M     0 100% /boot
tmpfs           6.3G     0  6.3G   0% /run/user/1001

As visible from the snippet above the /dev/md2 partition has 372G of space in use.

While investigating with du we realised that it was a container taking a lot of space and thus writing a massive file of about 330G under `/var/lib/docker/containers/fileID.log`  .

Although we removed the offending container and its whole stack from the Rancher deployment we realised the file was not being released and the disk space still allocated even if the file was deleted already. As a result we looked for deleted files that were not released yet. In order to look for it you can use lsof or in our case as it was not installed just use find/ls .

Indeed you can ls /proc/*/fd as such

sudo ls -lU /proc/*/fd | grep deleted 
lr-x------ 1 root root 64 Oct 13 2018 42 -> /var/lib/docker/containers/15ef8edcf7dcef2ea696fdef79e8b22150789227c86ec856570a49f086300e24/15ef8edcf7dcef2ea696fdef79e8b22150789227c86ec856570a49f086300e24-json.log (deleted)
lr-x------ 1 root root 64 Oct 14 2018 123 -> /var/lib/docker/containers/15ef8edcf7dcef2ea696fdef79e8b22150789227c86ec856570a49f086300e24/15ef8edcf7dcef2ea696fdef79e8b22150789227c86ec856570a49f086300e24-json.log (deleted)
lr-x------ 1 root root 64 Oct 15 2018 139 -> /var/lib/docker/containers/15ef8edcf7dcef2ea696fdef79e8b22150789227c86ec856570a49f086300e24/15ef8edcf7dcef2ea696fdef79e8b22150789227c86ec856570a49f086300e24-json.log (deleted)
lr-x------ 1 root root 64 Apr 18 06:00 146 -> /var/lib/docker/containers/15ef8edcf7dcef2ea696fdef79e8b22150789227c86ec856570a49f086300e24/15ef8edcf7dcef2ea696fdef79e8b22150789227c86ec856570a49f086300e24-json.log (deleted)

and see files marked as deleted but not released yet.

Then you have a few options, you can try to stop/restart the process blocking the file or as in our case, as this might cause a downtime to our client, we would just overwrite the file with empty content. If the file was still present it could be deleted with

: > /path/to/the/file.log

but as the file was yes deleted but still locked up by a process then we can overwrite it by looking up the process ID and the file descriptor and run the overwrite as shown here

: > "/proc/$pid/fd/$fd"

or

sudo sh -c ': > /proc/1233/fd/146'

if you experience permission problems in bash.

To find the process id and the file descriptor you can run

sudo find /proc/*/fd -ls | grep  deleted | grep docker
17288      0 lr-x------   1 root       root             64 Oct 13  2018 /proc/1233/fd/42 -> /var/lib/docker/containers/15ef8edcf7dcef2ea696fdef79e8b22150789227c86ec856570a49f086300e24/15ef8edcf7dcef2ea696fdef79e8b22150789227c86ec856570a49f086300e24-json.log\ (deleted)
1106161      0 lr-x------   1 root       root             64 Oct 14  2018 /proc/1233/fd/123 -> /var/lib/docker/containers/15ef8edcf7dcef2ea696fdef79e8b22150789227c86ec856570a49f086300e24/15ef8edcf7dcef2ea696fdef79e8b22150789227c86ec856570a49f086300e24-json.log\ (deleted)
4993234      0 lr-x------   1 root       root             64 Oct 15  2018 /proc/1233/fd/139 -> /var/lib/docker/containers/15ef8edcf7dcef2ea696fdef79e8b22150789227c86ec856570a49f086300e24/15ef8edcf7dcef2ea696fdef79e8b22150789227c86ec856570a49f086300e24-json.log\ (deleted)
1659260673      0 lr-x------   1 root       root             64 Apr 18 06:00 /proc/1233/fd/146 -> /var/lib/docker/containers/15ef8edcf7dcef2ea696fdef79e8b22150789227c86ec856570a49f086300e24/15ef8edcf7dcef2ea696fdef79e8b22150789227c86ec856570a49f086300e24-json.log\ (deleted)

and as you can see in bold above, the $pid and the $fd values will be visible in this breakdown.

Once you overwrite the content then your filesystem will finally be freed of some extra space.

$ df -h
Filesystem Size Used Avail Use% Mounted on
udev 32G 0 32G 0% /dev
tmpfs 6.3G 680M 5.7G 11% /run
/dev/md2 438G 38G 377G 10% /
tmpfs 32G 3.4M 32G 1% /dev/shm
tmpfs 5.0M 0 5.0M 0% /run/lock
tmpfs 32G 0 32G 0% /sys/fs/cgroup
/dev/md1 488M 486M 0 100% /boot
tmpfs 6.3G 0 6.3G 0% /run/user/1001

I hope you like this article, please feel free to share it online or contact us if you have any questions.

Gmail/GSuite and their hidden domain reputation factor, causing email connectivity damages to small businesses.

Major internet service provider causing extensive business damages to our business due to their unreasonable and hidden service policies.

As a small UK business we run online international software with several users signing up to our service from GMail or from email powered by GSuite, the Google email service provided to run email on their server on behalf of your website/business.

GMail is a great service but as a business we have discovered that they have an internal classification on how handle SPAM that goes as far as rejecting emails. An example of their rejection bounce message response shows

<d**************o@gmail.com>: host gmail-smtp-in.l.google.com[74.125.142.26] said: 550-5.7.1 [xxxxxxxxxxxxx] Our system has detected that this message is 550-5.7.1 likely suspicious due to the very low reputation of the sending 550-5.7.1 domain. To best protect our users from spam, the message has been 550-5.7.1 blocked. Please visit 550 5.7.1 https://support.google.com/mail/answer/188131 for more information. m14si485512pgs.39 – gsmtp (in reply to end of DATA command)

We have now experienced this disservice since the 29th of January 2020 and all our emails are rejected. This is despite following all their guidelines and their suggestion.

The situation is so bad that Google/Gmail users send emails to us and despite their interaction we are still unable to reply back because Gmail blocks all emails.

Below are a few facts/comments on our experience with GMail/GSuite while trying to troubleshoot the problem:

  1. Blocking emails is a really drastic measure and also causes damage to a business. Google tools and suggestion do not provide any clear evidence as to why emails are being blocked.
  2. Google tools suggested to troubleshoot the problem do not show any data why this the rejection/ban is in place
  3. Google does not provide any direct support. Instead they use informal indirect forums where product experts (likely to be Google employees) respond in a generic manner and do not provide any real resolution despite months of trying to troubleshoot this.
  4. Google knows their tools for monitoring traffic and help in the resolution of problems to not show any data, they do nothing to fix it so that the data is visible and show no other alternative on how to fix the problem. See an example of the several articles on their forum showing complaints about how “Postmaster” has no data https://support.google.com/mail/thread/4100957?hl=en . We do check on a daily basis and see no data at all but they still provide this as one of their tools to monitor email deliverability to their systems.
  5. Gmail/GSuite operate also as a premium provider, meaning that they also charge users to run emails on their servers and they are a major player in the market. Given this position can anyone suggest if their unreasonable blocking policy can be seen as an abusive position?
  6. Google product experts suggested to go through Troubleshooting for senders with email delivery issues and indeed we have none of the issues described in this form. The form leads to another form described in the next bullet point
  7. The Sender Contact Form is supposed to be a way to request that the ban/rejection is lifted. We have sent countless requests to this form with very detailed examples of how their service is bouncing our emails and still see the emails being rejected.

See our support thread on the Google forum with all the details exchanges with product experts and other users confirming all the steps we have taken into trying to troubleshoot the problem and how all the suggestions from them do not work https://support.google.com/mail/thread/27427166 .

If you are also experiencing this problem, managed to solve or you just want to contact us with more information regarding the problem please contact us.

GitLab Clone Private Repository with Access Token

When you are hosting private code, for instance for a reusable component, on Gitlab you will not be able to clone it unless you have access or are authenticated with the GitLab backend and are authorised to the repository.

Access tokens are a great way to allow an alternative way to clone or add your project as a dependency to a parent project. For example you can use tokens as part of your Continuous Integration pipeline to build, test and deploy your project.

If you add a project in your composer.json ( composer is the de facto package manager most used in PHP) , such as

composer.phar require "bizmate/my-private-package:2.*"

You will see something like

Could not fetch https://gitlab.com/api/v4/projects/bizmate%2Fmy-private-package/repository/archive.zip?sha=..., enter your gitlab.com credentials to access private repos
A token will be created and stored in "/home/composer/.composer/auth.json", your password will never be stored
To revoke access to this token you can visit gitlab.com/profile/applications
Username: 
Password: 
Bad credentials.
You can also manually create a personal token at https://gitlab.com/profile/personal_access_tokens
Add it using "composer config --global --auth gitlab-token.gitlab.com <token>"

Using username and password credentials is not a great approach as they are critical information and also because if you have 2FA enabled it might not work. So I created a personal access token and I tried adding it through the composer command but this did not work somehow. This might be due to the complexity also of running a development environment in docker but indeed I still found a simpler way to get around this.

Just add your access token to your git configuration, as per command below.

git config --global gitlab.accesstoken {TOKEN_VALUE}

The above will work straight away and you will be able to add your project in composer straight away without adding any information in its auth.json file.

Getting started with PHP and RabbitMQ queues using Docker in style

RabbitMQ is a great option for the implementation of an AMQP (Advanced Message Queuing Protocol) queuing system.

Why would you need a queuing system? So that you can offload heavy tasks to be processed in a separate process asynchronously and avoid blocking your client, on your website for instance.

How to perform multiple Guzzle requests at the same time?

Guzzle is a great wrapper to run Curl requests from your PHP applications

As part of my development requirements for MyReviews.link, I had to implement a fast concurrent way to perform http requests to several servers.