Upload a file to Azure Data Lake Store using .NET

Prerequisites

• An Azure subscription
• .NET Core SDK
• Visual Studio Code
• C# extension from the Visual Studio Code Marketplace

Contents

1. Register an Azure Application
2. Generate a Key (aka Client Secret)
3. Assign Permissions
4. Create a .NET Application
5. Install Dependencies
6. Code Sample

1. Register an Azure Application

1. Logon to the Azure Portal (https://portal.azure.com).
2. Navigate to Azure Active Directory > App registrations > New application registration.
   a. Name: Enter a name for your application (e.g. ADLS App).
   b. Application type: Select Web app / API as the application type.
   c. Sign-on URL: Enter a dummy sign-on URL (e.g. http://dummy).
3. Click Create.

2. Generate a Key (aka Client Secret)

1. Open your application (Azure Portal > Azure AD > App registrations > "Your Application").
2. Under "API Access" click Keys.
3. Under "Passwords" add an entry by populating:
   a. Key description: e.g. ADLS App Key
   b. Expiry: e.g. In 1 Year
4. Click Save.
5. Copy the key value to the clipboard and store it securely for later use. This key value will be used as the Client Secret in your .NET application.

Important: You won't be able to retrieve the key value after leaving the blade.

3. Assign Permissions

1. Open your Azure Data Lake Store resource (Azure Portal > All Resources > "Your Azure Data Lake Store").
2. Navigate to Overview > Data Explorer > Access.
3. Click Add.
4. Under "Select User or Group" find your application and click Select.
5. Assign permissions:
   a. Permissions: Check Read, Write and Execute.
   b. Add to: Check "This folder and all children".
   c. Add as: Check "An access permission entry".
6. Click OK.

4. Create a .NET Application

1. Open Visual Studio Code.
2. Select File > Open to open the folder you want your C# project to be in.
   Note: If the terminal is not active: View > Integrated Terminal
3. Initialise a C# project: dotnet new console
4. Run the program: dotnet run

If the text "Hello World!" was printed to the console, you have successfully created a .NET console application.

5. Install Dependencies

The code used in this example is dependent on several packages within the .NET Azure SDK. Instructions below on how the packages can be installed using the .NET Core CLI.

Execute the following commands into the terminal:

dotnet add package Microsoft.Azure.Management.DataLake.Store
dotnet add package Microsoft.IdentityModel.Clients.ActiveDirectory
dotnet add package Microsoft.Rest.ClientRuntime.Azure.Authentication

6. Code Sample

1. Create a source (input) text file in the same folder as your .NET application (e.g. source.txt).
2. Copy and paste the code below into Program.cs.
3. Update the following variables with your own values: clientId, clientSecret, tenantId & adlsAccountName.
4. Execute the following command into the terminal to run the program: dot net run

If the text "Finished!" has been printed to the console, you have successfully copied a text file from your local machine to the Azure Data Lake Store using the .NET SDK. To confirm, log on to the Azure portal and check that destination.txt exists in your Data Lake Store via Data Explorer.

using System;
using System.Threading;
using Microsoft.Azure.Management.DataLake.Store;
using Microsoft.IdentityModel.Clients.ActiveDirectory;
using Microsoft.Rest.Azure.Authentication;

namespace azure
{
    class Program
    {
        // Data Lake Store File System Management Client
        private static DataLakeStoreFileSystemManagementClient adlsFileSystemClient;

        // Portal > Azure AD > App Registrations > App > Application ID (aka Client ID)
        private static string clientId = "[ENTER_YOUR_APP_ID]";
                        
        // Portal > Azure AD > App Registrations > App > Settings > Keys (aka Client Secret)
        private static string clientSecret = "[ENTER_YOUR_APP_KEY]";

        // Portal > Azure AD > Properties > Directory ID (aka Tenant ID)
        private static string tenantId = "[ENTER_YOUR_TENANT_ID]";

        // Name of the Azure Data Lake Store
        private static string adlsAccountName = "[ENTER_YOUR_ADL_NAME]";

        private static void Main(string[] args)
        {
            // 1. Set Synchronization Context
            SynchronizationContext.SetSynchronizationContext(new SynchronizationContext());

            // 2. Create credentials to authenticate requests as an Active Directory application
            var clientCredential = new ClientCredential(clientId, clientSecret);
            var creds = ApplicationTokenProvider.LoginSilentAsync(tenantId, clientCredential).Result;

            // 2. Initialise Data Lake Store File System Client
            adlsFileSystemClient = new DataLakeStoreFileSystemManagementClient(creds);

            // 3. Upload a file to the Data Lake Store
            var source = "source.txt";
            var destination = "destination.txt";
            adlsFileSystemClient.FileSystem.UploadFile(adlsAccountName, source, destination, 1, false, true);

            // FINISHED
            Console.WriteLine("6. Finished!");
        }
    }
}