If you are coming from the Jenkins world, you may be wondering whether GitLab pipelines can accept parameters. They can and I’ll demonstrate below how to define and run parameterized pipelines in GitLab. If you later find this article useful take a look at the disclaimer for information on how to thank me.
Category: DevOps
Categories
Software Dependency Updates Using Renovate
Today, I’ll demo how to automate software dependency updates using Renovate tool. If you later find this article useful take a look at the disclaimer for information on how to thank me.
Categories
Go Docker CI in GitLab
Today, I’ll demo a sample Go Docker CI pipeline in GitLab. The pipeline will run on a sample Go containerized app. Hence the name Go Docker. If you later find this article useful take a look at the disclaimer for information on how to thank me.
Today, I’ll show how to keep git repository in another git repository. It may sound crazy and unnecessary. Yet, I’ve stumbled once on a need to do that during my career. So, I’ll share the way with the world. If you later find this article useful take a look at the disclaimer for information on how to thank me.
Why Store Git Repository in another Git Repository?
If you came across this article, you probably know why you need to do such a bizarre thing. Yet, for those who wonder why one might need to store a git repository inside another git repository, I’ll provide an example.
Suppose, your app needs to pull some code from a git repository (e.g. some script) and run it. So far, it seems like an easy use case. You test your app on a dev environment. Nicely, it has access to an org git repository hosting. Thus, there’s no issue to place your scripts in some git repository and provide repository url and credentials to your app. Now, suppose your app runs in an air-gapped environment for security reasons. In that case, the app obviously won’t have access to git repository hosting and must use a different solution. You might say, why require git access in the first place. It’s a rightful question for the apps you fully control. Yet, what if you use some 3rd party product which you cannot modify? I stumbled on such a use case while using AWX.
Real world use-case requiring storing Git Repository in another Git Repository
Simply put, AWX is a web app for managing ansible playbooks. Ansible requires an inventory of hosts to run the playbooks on. And such an inventory may come from a dynamic inventory script. AWX pulls this script from a git repository. That’s how it works and I can do nothing about it. Now imagine, AWX running in production air-gapped environment. How would you provide access from AWX to a git repository? Short things short, I came up with the solution of keeping an inventory script inside a bare git repository which resided in the code repository of AWX out of which its deployable artifact was built. Let’s see git commands I used to create bare git repository inside another repository.
How to store Git Repository in another Git Repository?
To achieve that, we’ll need to store the inner repository as files. This way, it can be source controlled just as any other data you want to store in git. Follow below commands to store repository (inner below) inside another git repository (outer below):
- Create outer git repository:
mkdir /tmp/outer
cd /tmp/outer
git init
echo "i'm outer" > outer.txt
git add *
git commit -am 'outer'- Create inner git repository:
mkdir /tmp/inner
cd /tmp/inner
git init- make sample change inside
inner:
echo "i'm inner" > inner.text
git add inner.text
git commit -am 'test'- Copy .git contents of
innerrepo toouter
mkdir /tmp/outer/inner
cp -r /tmp/inner/.git/* /tmp/outer/inner Now, commit inner inside outer repository:
cd /tmp/outer
git add *
git commit -m 'inner inside outer'That’s it! You can now push outer repository, build a deployable artifact out of it and deploy it to production. Anyone or anything will be able to clone inner repository to get its contents. Try it locally:
git clone /tmp/outer/inner /tmp/inner_contents
cat /tmp/inner_contents/inner.text
i'm innerIf you followed to this point, then you might really needed this to work as I did 🙂
Summary
That’s it about keeping git repository in another git repository. Feel free to share.
If you found this article useful, take a look at the disclaimer for information on how to thank me.
You can find below articles useful as well:
Today, I’ll show how to run a sample Terraform pipeline on GitLab. The pipeline will include GitLab Terraform CI templates. If you later find this article useful take a look at the disclaimer for information on how to thank me.
Assume you want to clone a private GitLab repository in your GitLab pipeline. Let’s see how to do that. If you later find this article useful take a look at the disclaimer for information on how to thank me.
Why Clone Repositories in GitLab Pipelines?
You probably know why you need that. You have never done it and wonder why you may need it? For example, your GitLab pipeline may run some automation (e.g. patterns replacements, version bumps) on specific repositories. To perform the changes in the repositories, you first need to clone the repository in your pipeline, do some changes, then commit and push it. However first, you need to clone it. While cloning repositories manually is straightforward either using https or ssh url, cloning in GitLab pipelines requires a bit different url. Let’s see how it looks and a demo GitLab pipeline using it.
Clone Private Repositories in GitLab Pipelines Demo
Let’s have a look at a sample GitLab pipeline which clones another private repository. In addition, it makes a sample change, commits and pushes it. Note that cloning private repository requires authentication, while clone public does not. To authenticate you’d need to generate a personal or group access token of at least read_repository scope. Add the token to the GitLab project or group masked variables.
Below is the pipeline which uses the variable GITLAB_TOKEN.
stages:
- update_repo
build-job:
stage: update_repo
- git config --global user.email "[email protected]"
- git config --global user.name "Your Name"
script:
- echo "updating repo"
- git clone https://oauth2:[email protected]/[your_user]/[another_project_name].git
- cd [project_name]
- echo 'test' > test.txt
- git commit -am 'test'
- git push
- cd -Note above that to clone the repository you need to run:
git clone https://oauth2:[email protected]/[your_user]/[another_project_name].git where GITLAB_TOKEN is defined as the GitLab project or group masked variable.
Summary
That’s it about using clone private repositories in GitLab pipelines. As always feel free to share. If you found this article useful, take a look at the disclaimer for information on how to thank me.
You can also find below articles useful:
- GitLab Self-Hosted Runners Demo
- Get user’s permissions using kubectl
- Migration from Jenkins to GitLab
- Git Tricks: git commit –amend + git force –push
Recommended GitLab books on Amazon.
Categories
Use Ansible Vault in Python
So you want to use secrets stored in Ansible Vault in your Python apps. Let’s see how to do that. If you later find this article useful take a look at the disclaimer for information on how to thank me.
If you are not familiar with Ansible Vault, go over the brief introduction below.
Store secrets in Ansible Vault
So, you might already know that storing secrets in your source code is bad. Yet, your app uses secrets and must store them in source code repository. What can you do? You can opt for a fully fledged secret storage solution (e.g. HashiCorp Vault). Yet, this is an overkill for a simple app using secrets. What can you do? The simplest thing that comes to mind is encrypting the secrets and keeping them encrypted under source control. Ansible Vault allows just that.
Storing secrets in Ansible Vault step by step
- Create
vault.ymlfile and add your secrets in yaml format as below:
secret_name1: val1
secret_name2: val2- Next, create
multi_password_file. Add the password to the vault to the file and add it to.gitignore. This is the password which will encrypt the vault. Remember that if you lose this password file, you won’t be able to decrypt your Ansible vault. - Install
ansible. Installing it will installansible-vaultbinary as well and add it to your PATH. - Run
ansible-vault encrypt vault.yml --vault-password-file multi_password_fileto encrypt your vault. Afterwards,vault.ymlwill start with$ANSIBLE_VAULT;1.1;AES256and will contain just numbers. - Run
ansible-vault decrypt vault.yml --vault-password-file multi_password_fileto decrypt the vault. Then you’ll see your secrets in clear text. You can safely commitvault.ymlto source code repository.
Read Ansible Vault in Python
Now, let’s assume you want to use the secrets from Ansible Vault in your Python app or script. How can you read it? You can do that using ansible-vault package. Then use below Python code for reading the vault:
from pathlib import Path
from ansible_vault import Vault
vault = Vault((Path('multi_password_file').read_text()))
data = vault.load(open('vault.yml').read())
data is Python dictionary which contains the Ansible vault secrets in clear text which your app/script can use.
Summary
That’s it about using Ansible Vault in Python. As always feel free to share. If you found this article useful, take a look at the disclaimer for information on how to thank me.
You can also find below articles useful:
- Azure-cli in Dockerfile in Alpine
- Podman Jenkins Agent
- Go Docker CI in GitLab
- GitLab Parameterized Pipelines
Recommended Kubernetes books on Amazon:
Categories
Helm charts acceptance tests
I already covered how to test helm charts and different tests you may want to run. Today, I’ll focus on helm charts acceptance tests. If you later find this article useful take a look at the disclaimer for information on how to thank me.
Today, I’ll demo Kafka producer and consumer written in Python. We’ll see a fully working demo of producer and consumer running against Kafka in a docker-compose stack.
If you later find this article useful take a look at the disclaimer for information on how to thank me.
If you developed modern CI/CD pipelines you probably stumbled on the need to auto tag releases with semantic versions. Today I’ll show how to do that automatically and which tools may help to achieve automatic tagging releases with semantic versions.
If you later find this article useful take a look at the disclaimer for information on how to thank me.