Posts

CI/CD pipeline for data movement from Azure Repository to ADLS

Image
Author: Akhil M Anil || DevOps Engineer Azure Repos is a cloud-based version control system that helps organizations manage and collaborate on their code and other files. It is an essential tool for software development teams to keep track of changes and maintain a history of their codebase. On the other hand, Azure ADLS is a highly scalable and secure data lake storage solution that enables organizations to store, manage, and analyze large amounts of structured and unstructured data. It is a cost-effective solution that can store and process data at any scale, making it an ideal option for big data analytics and machine learning workloads. In this blog, we will discuss how to move data from Azure Repos to Azure ADLS. We will explore the different methods and tools available for data movement and the benefits of using Azure ADLS as a data storage solution. Additionally, we will also discuss the best practices for data migration and how to ensure data integrity during the process. Step
Post a Comment
Read more

List of Repositories - Authorize Rest API calls using Job Access Token

Image
  Author: Akhil M Anil || DevOps Engineer Getting the complete list of Repository names  in Azure DevOps Project using REST Api and Publishing as a CSV file. In the last blog I have talked about Authorizing REST API call using Job Access Tokens. If missed, please find that blog here:  Job Access Tokens . In this blog we will use Job Access Token to Authorize Rest API call and retrieve the list of repositories.  Full code can be found in GitHub:  Repository-List Steps: Push the developed pipeline code  (azure-pipelines-repos.yml) and python code  (repos.py) to Azure Repos. We are using  Repositories - List - REST API  to fetch the repository list using python. Job Authorization Scope for Job Access token. In order to access repositories using system access token, we need to change our default job authorization scope. In my last blog I have already mentioned about Job Authorization Scopes, feel free to take a look.  To set job authorization scope at the organization level for all projec
Post a Comment
Read more

Job Access Tokens || Authorize REST API calls to access resources in ADO using Job Access Tokens

Image
  Author: Akhil M Anil || DevOps Engineer For applications that interface with Azure DevOps Services, you must authenticate to gain access to resources like REST APIs. There are different ways to authenticate our applications with Azure DevOps Services. In this article we will use job access token/system access token (SYS_TOKEN) to authenticate into Azure DevOps. A job access token is a security token that is dynamically generated by Azure Pipelines for each job at run time. The agent on which the job is running uses the job access token in order to access these resources in Azure DevOps. It is fully possible to use System Access Tokens and eliminate the need for PATs in ADO pipeline for authorize the REST API calls to ADO. Furthermore, there is no need to save any credentials to files on disc and the System Access Tokens can be leveraged transiently as environment variables scoped to a specific job. The token's permissions are derived from job authorization scope the permissions
Post a Comment
Read more

Authenticate to Azure DevOps using PAT token

Image
  Author: Akhil M Anil || DevOps Engineer For applications that interface with Azure DevOps Services, you must authenticate to gain access to resources like REST APIs. There are different ways to authenticate our applications with Azure DevOps Services. In this article we will use a personal access token (PAT) to authenticate into Azure DevOps. A personal access token contains your security credentials for Azure DevOps. A PAT identifies you, your accessible organizations, and scopes of access. As such, they're as critical as passwords. Since PAT is as identical as a user credential many organizations restrict its usage in Azure DevOps Api's instead promote the usage of any other authentication method or store the PAT token in Key-Vaults and use them. 1. Let's create a PAT token in ADO.  Sign into your organization From the home screen select the user settings and select Personal Access Token. Select +New Token. Give a name for the token, select the organization then set y
Post a Comment
Read more

Usage of Variable Template in Azure DevOps Pipelines

Image
  Author: Akhil M Anil || DevOps Engineer Let's understand what variables in Azure DevOps are:  Variables give us a convenient way to get key bits of data into various parts of the pipeline. The most common use of variables is to define a value that we can then use in our pipeline. The value of a variable can change from run to run or job to job in your pipeline. Different types of variables are: User-defined variables System variables Environment variables User-defined Variables The variables defined by the user are called user-defined variables. In YAML pipelines, we can set variables at the root, stage, and job levels. We can use a variable group to make variables available across multiple pipelines or we can use templates to define variables that are used in multiple pipelines in one file. In this blog, we will discuss variables defined in a template file and create a simple pipeline for getting variables from the variable template.  Advantages of using Variable Templates over
Post a Comment
Read more

Use an Azure Virtual Machine Scale set as Azure DevOps agents

Image
Author: Akhil M Anil || DevOps Engineer An agent in  Azure DevOps is a computing infrastructure with installed agent software that runs a job to perform a task/ a set of tasks. There are different types of agents available for the users such as: Microsoft Hosted Agents Self-hosted Agents Scale Set Agents Azure Virtual Machine Scale Set Agents Azure Virtual Machine Scale Set agents is a feature within Azure DevOps. It allows you to use an existing Virtual Machine Scale Set to host your Azure DevOps agents and allows Azure DevOps to manage the scaling out and in of those agents. Therefore, you get the ability to deploy your agents in a private network and control the dependencies that are installed on the agents. Note: You cannot run Mac agents using scale sets. You can only run Windows or Linux agents this way. Scale set agents currently supports Ubuntu Linux, Windows Server/DataCenter 2016/2019, and Windows 10 client. In this article, we will configure the Linux Scale Set agent, insta
Post a Comment
Read more

Configure an Azure DevOps self-hosted Windows agent in Docker

Image
Author: Akhil M Anil || DevOps Engineer An agent is computing infrastructure with installed agent software that runs job to perform a task/ a set of tasks. There are different types of agents available for the users such as: Microsoft Hosted Agents Self-hosted Agents Scale Set Agents Self-hosted Agents An agent that we set up and manage on our own to run jobs is a  self-hosted agent .  Self-hosted agents give us more control to install dependent software needed for our builds and deployments. Also, machine-level caches and configuration persist from run to run, which can boost speed. We  can install the agent on Linux, macOS, or Windows machines. We can also install an agent on a Docker container. This Article covers configuring an agent on a Docker container and use the agent to run job to perform tasks. Run a self-hosted agent in Docker We can set up a self-hosted agent in Azure Pipelines to run inside a Windows Server Core (for Windows hosts), or Ubuntu container (for Linux hosts)
Post a Comment
Read more