Menggunakan Fungsi Awan Google untuk Pemprosesan Data Tiga Peringkat dengan Komposer Google dan Penyerahan Automatik melalui Tindakan GitHub-Tutorial Python-php.cn

Jadual Kandungan

Mengenai
Alat Digunakan
Rajah Seni Bina Penyelesaian
Proses Penyerahan
- Prasyarat
- Rahsia untuk Tindakan GitHub
- Untuk mencipta rahsia baharu:
- Menyediakan Akaun Perkhidmatan DevOps
Talian Paip Tindakan GitHub: Langkah
- dayakan-perkhidmatan
- terapkan baldi
- fungsi-cloud-deploy
- deploy-composer-service-account
- deploy-bigquery-dataset-bigquery-tables
- tetapkan-persekitaran-komposer
- deploy-composer-http-connection
- deploy-dags
Penjelasan Aliran Kerja Tindakan GitHub
Sumber Dicipta Selepas Penggunaan
Kesimpulan
Rujukan

Tentang

Siaran ini meneroka penggunaan Fungsi Awan Google untuk memproses data dalam seni bina tiga peringkat. Penyelesaiannya diatur dengan Google Composer dan menampilkan penyerahan automatik menggunakan GitHub Actions. Kami akan menelusuri alatan yang digunakan, proses pelaksanaan dan langkah saluran paip, menyediakan panduan yang jelas untuk membina saluran paip data berasaskan awan hujung ke hujung.

Alatan Digunakan

Google Cloud Platform (GCP): Persekitaran awan utama.
Cloud Storage: Untuk menyimpan input dan data yang diproses merentas lapisan berbeza (Gangsa, Perak, Emas).
Fungsi Cloud: Fungsi tanpa pelayan yang bertanggungjawab untuk pemprosesan data dalam setiap peringkat.
Komposer Google: Alat orkestra berdasarkan Aliran Udara Apache, digunakan untuk menjadualkan dan mengurus aliran kerja.
Tindakan GitHub: Alat automasi untuk mengatur dan mengurus infrastruktur saluran paip.

Rajah Seni Bina Penyelesaian

Using Google Cloud Functions for Three-Tier Data Processing with Google Composer and Automated Deployments via GitHub Actions

Proses Penyerahan

Prasyarat

Sebelum menyediakan projek, pastikan anda mempunyai perkara berikut:

Akaun GCP: Akaun Google Cloud dengan pengebilan didayakan.
Akaun Perkhidmatan untuk DevOps: Akaun perkhidmatan dengan kebenaran yang diperlukan untuk menggunakan sumber dalam GCP.
Rahsia dalam GitHub: Simpan bukti kelayakan akaun perkhidmatan GCP, ID projek dan nama baldi sebagai rahsia dalam GitHub untuk akses selamat.

Rahsia untuk Tindakan GitHub

Untuk mengakses projek dan sumber GCP anda dengan selamat, tetapkan rahsia berikut dalam Tindakan GitHub:

BUCKET_DATALAKE: Baldi Storan Awan anda untuk tasik data.
GCP_DEVOPS_SA_KEY: Kunci akaun perkhidmatan dalam format JSON.
PROJECT_ID: ID projek GCP anda.
REGION_PROJECT_ID: Rantau tempat projek GCP anda digunakan.

Untuk mencipta rahsia baharu:

1. In project repository, menu **Settings** 
2. **Security**, 
3. **Secrets and variables**,click in access **Action**
4. **New repository secret**, type a **name** and **value** for secret.

Salin selepas log masuk

Using Google Cloud Functions for Three-Tier Data Processing with Google Composer and Automated Deployments via GitHub Actions

Untuk butiran lanjut, akses :
https://docs.github.com/pt/actions/security-for-github-actions/security-guides/using-secrets-in-github-actions

Menyediakan Akaun Perkhidmatan DevOps

Buat akaun perkhidmatan dalam GCP dengan kebenaran untuk Cloud Functions, Composer, BigQuery dan Cloud Storage. Berikan peranan yang diperlukan seperti:

Pentadbir Fungsi Cloud
Pengguna Komposer
Editor Data BigQuery
Pentadbir Objek Storan

Talian Paip Tindakan GitHub: Langkah

Saluran paip mengautomasikan keseluruhan proses penggunaan, memastikan semua komponen disediakan dengan betul. Berikut ialah pecahan tugas utama daripada fail GitHub Actions, masing-masing bertanggungjawab untuk aspek penempatan yang berbeza.

enable-services

enable-services:
    runs-on: ubuntu-22.04
    steps:
    - uses: actions/checkout@v2

    # Step to Authenticate with GCP
    - name: Authorize GCP
      uses: 'google-github-actions/auth@v2'
      with:
        credentials_json:  ${{ secrets.GCP_DEVOPS_SA_KEY }}

    # Step to Configure  Cloud SDK
    - name: Set up Cloud SDK
      uses: google-github-actions/setup-gcloud@v2
      with:
        version: '>= 363.0.0'
        project_id: ${{ secrets.PROJECT_ID }}

    # Step to Configure Docker to use the gcloud command-line tool as a credential helper
    - name: Configure Docker
      run: |-
        gcloud auth configure-docker

    - name: Set up python 3.8
      uses: actions/setup-python@v2
      with:
        python-version: 3.8.16

    # Step to Create GCP Bucket 
    - name: Enable gcp service api's
      run: |-
        gcloud services enable ${{ env.GCP_SERVICE_API_0 }}
        gcloud services enable ${{ env.GCP_SERVICE_API_1 }}
        gcloud services enable ${{ env.GCP_SERVICE_API_2 }}
        gcloud services enable ${{ env.GCP_SERVICE_API_3 }}
        gcloud services enable ${{ env.GCP_SERVICE_API_4 }}

Salin selepas log masuk

deploy-baldi

deploy-buckets:
    needs: [enable-services]
    runs-on: ubuntu-22.04
    timeout-minutes: 10

    steps:
    - name: Checkout
      uses: actions/checkout@v4

    - name: Authorize GCP
      uses: 'google-github-actions/auth@v2'
      with:
        credentials_json:  ${{ secrets.GCP_DEVOPS_SA_KEY }}

    # Step to Authenticate with GCP
    - name: Set up Cloud SDK
      uses: google-github-actions/setup-gcloud@v2
      with:
        version: '>= 363.0.0'
        project_id: ${{ secrets.PROJECT_ID }}

    # Step to Configure Docker to use the gcloud command-line tool as a credential helper
    - name: Configure Docker
      run: |-
        gcloud auth configure-docker

    # Step to Create GCP Bucket 
    - name: Create Google Cloud Storage - datalake
      run: |-
        if ! gsutil ls -p ${{ secrets.PROJECT_ID }} gs://${{ secrets.BUCKET_DATALAKE }} &> /dev/null; \
          then \
            gcloud storage buckets create gs://${{ secrets.BUCKET_DATALAKE }} --default-storage-class=nearline --location=${{ env.REGION }}
          else
            echo "Cloud Storage : gs://${{ secrets.BUCKET_DATALAKE }}  already exists" ! 
          fi


    # Step to Upload the file to GCP Bucket - transient files
    - name: Upload transient files to Google Cloud Storage
      run: |-
        TARGET=${{ env.INPUT_FOLDER }}
        BUCKET_PATH=${{ secrets.BUCKET_DATALAKE }}/${{ env.INPUT_FOLDER }}    
        gsutil cp -r $TARGET gs://${BUCKET_PATH}

Salin selepas log masuk

deploy-cloud-function

deploy-cloud-function:
    needs: [enable-services, deploy-buckets]
    runs-on: ubuntu-22.04
    steps:
    - uses: actions/checkout@v2

    # Step to Authenticate with GCP
    - name: Authorize GCP
      uses: 'google-github-actions/auth@v2'
      with:
        credentials_json:  ${{ secrets.GCP_DEVOPS_SA_KEY }}

    # Step to Configure  Cloud SDK
    - name: Set up Cloud SDK
      uses: google-github-actions/setup-gcloud@v2
      with:
        version: '>= 363.0.0'
        project_id: ${{ secrets.PROJECT_ID }}

    # Step to Configure Docker to use the gcloud command-line tool as a credential helper
    - name: Configure Docker
      run: |-
        gcloud auth configure-docker

    - name: Set up python 3.10
      uses: actions/setup-python@v2
      with:
        python-version: 3.10.12
    #cloud_function_scripts/csv_to_parquet
    - name: Create cloud function - ${{ env.CLOUD_FUNCTION_1_NAME }}
      run: |-
        cd ${{ env.FUNCTION_SCRIPTS }}/${{ env.CLOUD_FUNCTION_1_NAME }}
        gcloud functions deploy ${{ env.CLOUD_FUNCTION_1_NAME }} \
        --gen2 \
        --cpu=${{ env.FUNCTION_CPU  }} \
        --memory=${{ env.FUNCTION_MEMORY  }} \
        --runtime ${{ env.PYTHON_FUNCTION_RUNTIME }} \
        --trigger-http \
        --region ${{ env.REGION }} \
        --entry-point ${{ env.CLOUD_FUNCTION_1_NAME }}

    - name: Create cloud function - ${{ env.CLOUD_FUNCTION_2_NAME }}
      run: |-
        cd ${{ env.FUNCTION_SCRIPTS }}/${{ env.CLOUD_FUNCTION_2_NAME }}
        gcloud functions deploy ${{ env.CLOUD_FUNCTION_2_NAME }} \
        --gen2 \
        --cpu=${{ env.FUNCTION_CPU  }} \
        --memory=${{ env.FUNCTION_MEMORY  }} \
        --runtime ${{ env.PYTHON_FUNCTION_RUNTIME }} \
        --trigger-http \
        --region ${{ env.REGION }} \
        --entry-point ${{ env.CLOUD_FUNCTION_2_NAME }}

    - name: Create cloud function - ${{ env.CLOUD_FUNCTION_3_NAME }}
      run: |-
        cd ${{ env.FUNCTION_SCRIPTS }}/${{ env.CLOUD_FUNCTION_3_NAME }}
        gcloud functions deploy ${{ env.CLOUD_FUNCTION_3_NAME }} \
        --gen2 \
        --cpu=${{ env.FUNCTION_CPU  }} \
        --memory=${{ env.FUNCTION_MEMORY  }} \
        --runtime ${{ env.PYTHON_FUNCTION_RUNTIME }} \
        --trigger-http \
        --region ${{ env.REGION }} \
        --entry-point ${{ env.CLOUD_FUNCTION_3_NAME }}

Salin selepas log masuk

deploy-composer-service-account

deploy-composer-service-account:
    needs: [enable-services, deploy-buckets, deploy-cloud-function ]
    runs-on: ubuntu-22.04
    timeout-minutes: 10

    steps:
    - name: Checkout
      uses: actions/checkout@v4

    - name: Authorize GCP
      uses: 'google-github-actions/auth@v2'
      with:
        credentials_json:  ${{ secrets.GCP_DEVOPS_SA_KEY }}

    # Step to Authenticate with GCP
    - name: Set up Cloud SDK
      uses: google-github-actions/setup-gcloud@v2
      with:
        version: '>= 363.0.0'
        project_id: ${{ secrets.PROJECT_ID }}

    # Step to Configure Docker to use the gcloud command-line tool as a credential helper
    - name: Configure Docker
      run: |-
        gcloud auth configure-docker


    - name: Create service account
      run: |-

        if ! gcloud iam service-accounts list | grep -i ${{ env.SERVICE_ACCOUNT_NAME}} &> /dev/null; \
          then \
            gcloud iam service-accounts create ${{ env.SERVICE_ACCOUNT_NAME }} \
            --display-name=${{ env.SERVICE_ACCOUNT_DESCRIPTION }}
          fi
    - name: Add permissions to service account
      run: |-
        gcloud projects add-iam-policy-binding ${{secrets.PROJECT_ID}} \
          --member="serviceAccount:${{env.SERVICE_ACCOUNT_NAME}}@${{secrets.PROJECT_ID}}.iam.gserviceaccount.com" \
          --role="roles/composer.user"

        gcloud projects add-iam-policy-binding ${{secrets.PROJECT_ID}} \
          --member="serviceAccount:${{env.SERVICE_ACCOUNT_NAME}}@${{secrets.PROJECT_ID}}.iam.gserviceaccount.com" \
          --role="roles/storage.objectAdmin"

        gcloud projects add-iam-policy-binding ${{secrets.PROJECT_ID}} \
          --member="serviceAccount:${{env.SERVICE_ACCOUNT_NAME}}@${{secrets.PROJECT_ID}}.iam.gserviceaccount.com" \
          --role="roles/cloudfunctions.invoker"

        # Permissão para criar e gerenciar ambientes Composer
        gcloud projects add-iam-policy-binding ${{secrets.PROJECT_ID}} \
          --member="serviceAccount:${{env.SERVICE_ACCOUNT_NAME}}@${{secrets.PROJECT_ID}}.iam.gserviceaccount.com" \
          --role="roles/composer.admin"

        gcloud projects add-iam-policy-binding ${{secrets.PROJECT_ID}} \
          --member="serviceAccount:${{env.SERVICE_ACCOUNT_NAME}}@${{secrets.PROJECT_ID}}.iam.gserviceaccount.com" \
          --role="roles/composer.worker"

        # Permissão para criar e configurar instâncias e recursos na VPC
        gcloud projects add-iam-policy-binding ${{secrets.PROJECT_ID}} \
          --member="serviceAccount:${{env.SERVICE_ACCOUNT_NAME}}@${{secrets.PROJECT_ID}}.iam.gserviceaccount.com" \
          --role="roles/compute.networkAdmin"

        # Permissão para interagir com o Cloud Storage, necessário para buckets e logs
        gcloud projects add-iam-policy-binding ${{secrets.PROJECT_ID}} \
          --member="serviceAccount:${{env.SERVICE_ACCOUNT_NAME}}@${{secrets.PROJECT_ID}}.iam.gserviceaccount.com" \
          --role="roles/storage.admin"

        # Permissão para criar e gerenciar recursos no projeto, como buckets e instâncias
        gcloud projects add-iam-policy-binding ${{secrets.PROJECT_ID}} \
          --member="serviceAccount:${{env.SERVICE_ACCOUNT_NAME}}@${{secrets.PROJECT_ID}}.iam.gserviceaccount.com" \
          --role="roles/editor"

        # Permissão para acessar e usar recursos necessários para o IAM
        gcloud projects add-iam-policy-binding ${{secrets.PROJECT_ID}} \
          --member="serviceAccount:${{env.SERVICE_ACCOUNT_NAME}}@${{secrets.PROJECT_ID}}.iam.gserviceaccount.com" \
          --role="roles/iam.serviceAccountUser"

        gcloud functions add-iam-policy-binding ${{env.CLOUD_FUNCTION_1_NAME}} \
          --region="${{env.REGION}}" \
          --member="serviceAccount:${{env.SERVICE_ACCOUNT_NAME}}@${{secrets.PROJECT_ID}}.iam.gserviceaccount.com" \
          --role="roles/cloudfunctions.invoker"

        gcloud functions add-invoker-policy-binding ${{env.CLOUD_FUNCTION_1_NAME}} \
          --region="${{env.REGION}}" \
          --member="serviceAccount:${{env.SERVICE_ACCOUNT_NAME}}@${{secrets.PROJECT_ID}}.iam.gserviceaccount.com" 

        gcloud functions add-iam-policy-binding ${{env.CLOUD_FUNCTION_2_NAME}} \
          --region="${{env.REGION}}" \
          --member="serviceAccount:${{env.SERVICE_ACCOUNT_NAME}}@${{secrets.PROJECT_ID}}.iam.gserviceaccount.com" \
          --role="roles/cloudfunctions.invoker"

        gcloud functions add-invoker-policy-binding ${{env.CLOUD_FUNCTION_2_NAME}} \
          --region="${{env.REGION}}" \
          --member="serviceAccount:${{env.SERVICE_ACCOUNT_NAME}}@${{secrets.PROJECT_ID}}.iam.gserviceaccount.com"     

        gcloud functions add-iam-policy-binding ${{env.CLOUD_FUNCTION_3_NAME}} \
          --region="${{env.REGION}}" \
          --member="serviceAccount:${{env.SERVICE_ACCOUNT_NAME}}@${{secrets.PROJECT_ID}}.iam.gserviceaccount.com" \
          --role="roles/cloudfunctions.invoker"

        gcloud functions add-invoker-policy-binding ${{env.CLOUD_FUNCTION_3_NAME}} \
          --region="${{env.REGION}}" \
          --member="serviceAccount:${{env.SERVICE_ACCOUNT_NAME}}@${{secrets.PROJECT_ID}}.iam.gserviceaccount.com"   

        SERVICE_NAME_1=$(gcloud functions describe ${{ env.CLOUD_FUNCTION_1_NAME }} --region=${{ env.REGION }} --format="value(serviceConfig.service)")
        gcloud run services add-iam-policy-binding $SERVICE_NAME_1 \
        --region="${{env.REGION}}" \
        --member="serviceAccount:${{env.SERVICE_ACCOUNT_NAME}}@${{secrets.PROJECT_ID}}.iam.gserviceaccount.com" \
        --role="roles/run.invoker"

        SERVICE_NAME_2=$(gcloud functions describe ${{ env.CLOUD_FUNCTION_2_NAME }} --region=${{ env.REGION }} --format="value(serviceConfig.service)")
        gcloud run services add-iam-policy-binding $SERVICE_NAME_2 \
        --region="${{env.REGION}}" \
        --member="serviceAccount:${{env.SERVICE_ACCOUNT_NAME}}@${{secrets.PROJECT_ID}}.iam.gserviceaccount.com" \
        --role="roles/run.invoker"

        SERVICE_NAME_3=$(gcloud functions describe ${{ env.CLOUD_FUNCTION_3_NAME }} --region=${{ env.REGION }} --format="value(serviceConfig.service)")
        gcloud run services add-iam-policy-binding $SERVICE_NAME_3 \
        --region="${{env.REGION}}" \
        --member="serviceAccount:${{env.SERVICE_ACCOUNT_NAME}}@${{secrets.PROJECT_ID}}.iam.gserviceaccount.com" \
        --role="roles/run.invoker"


        gcloud functions add-invoker-policy-binding ${{env.CLOUD_FUNCTION_1_NAME}} \
        --region="${{env.REGION}}" \
        --member="allUsers"

        gcloud functions add-invoker-policy-binding ${{env.CLOUD_FUNCTION_2_NAME}} \
        --region="${{env.REGION}}" \
        --member="allUsers"

        gcloud functions add-invoker-policy-binding ${{env.CLOUD_FUNCTION_3_NAME}} \
        --region="${{env.REGION}}" \
        --member="allUsers"

Salin selepas log masuk

deploy-bigquery-dataset-bigquery-tables

deploy-bigquery-dataset-bigquery-tables:
    needs: [enable-services, deploy-buckets, deploy-cloud-function, deploy-composer-service-account ]
    runs-on: ubuntu-22.04
    timeout-minutes: 10

    steps:
    - name: Checkout
      uses: actions/checkout@v4

    - name: Authorize GCP
      uses: 'google-github-actions/auth@v2'
      with:
        credentials_json:  ${{ secrets.GCP_DEVOPS_SA_KEY }}

    # Step to Authenticate with GCP
    - name: Set up Cloud SDK
      uses: google-github-actions/setup-gcloud@v2
      with:
        version: '>= 363.0.0'
        project_id: ${{ secrets.PROJECT_ID }}

    # Step to Configure Docker to use the gcloud command-line tool as a credential helper
    - name: Configure Docker
      run: |-
        gcloud auth configure-docker


    - name: Create Big Query Dataset
      run: |-  
        if ! bq ls --project_id ${{ secrets.PROJECT_ID}} -a | grep -w ${{ env.BIGQUERY_DATASET}} &> /dev/null; \
          then 

            bq --location=${{ env.REGION }} mk \
          --default_table_expiration 0 \
          --dataset ${{ env.BIGQUERY_DATASET }}

          else
            echo "Big Query Dataset : ${{ env.BIGQUERY_DATASET }} already exists" ! 
          fi

    - name: Create Big Query table
      run: |-
        TABLE_NAME_CUSTOMER=${{ env.BIGQUERY_DATASET}}.${{ env.BIGQUERY_TABLE_CUSTOMER}}
        c=0
        for table in $(bq ls --max_results 1000 "${{ secrets.PROJECT_ID}}:${{ env.BIGQUERY_DATASET}}" | tail -n +3 | awk '{print }'); do

            # Determine the table type and file extension
            if bq show --format=prettyjson $BIGQUERY_TABLE_CUSTOMER | jq -r '.type' | grep -q -E "TABLE"; then
              echo "Dataset ${{ env.BIGQUERY_DATASET}} already has table named : $table " !
              if [ "$table" == "${{ env.BIGQUERY_TABLE_CUSTOMER}}" ]; then
                echo "Dataset ${{ env.BIGQUERY_DATASET}} already has table named : $table " !
                ((c=c+1))       
              fi                  
            else
                echo "Ignoring $table"            
                continue
            fi
        done
        echo " contador $c "
        if [ $c == 0 ]; then
          echo "Creating table named : $table for Dataset ${{ env.BIGQUERY_DATASET}} " !

          bq mk --table \
          $TABLE_NAME_CUSTOMER \
          ./big_query_schemas/customer_schema.json


        fi

Salin selepas log masuk

deploy-composer-environment

deploy-composer-environment:
    needs: [enable-services, deploy-buckets, deploy-cloud-function, deploy-composer-service-account, deploy-bigquery-dataset-bigquery-tables ]
    runs-on: ubuntu-22.04
    timeout-minutes: 40

    steps:
    - name: Checkout
      uses: actions/checkout@v4

    - name: Authorize GCP
      uses: 'google-github-actions/auth@v2'
      with:
        credentials_json:  ${{ secrets.GCP_DEVOPS_SA_KEY }}

    # Step to Authenticate with GCP
    - name: Set up Cloud SDK
      uses: google-github-actions/setup-gcloud@v2
      with:
        version: '>= 363.0.0'
        project_id: ${{ secrets.PROJECT_ID }}

    # Step to Configure Docker to use the gcloud command-line tool as a credential helper
    - name: Configure Docker
      run: |-
        gcloud auth configure-docker

    # Create composer environments
    - name: Create composer environments
      run: |-
        if ! gcloud composer environments list --project=${{ secrets.PROJECT_ID }} --locations=${{ env.REGION }} | grep -i ${{ env.COMPOSER_ENV_NAME }} &> /dev/null; then
            gcloud composer environments create ${{ env.COMPOSER_ENV_NAME }} \
                --project ${{ secrets.PROJECT_ID }} \
                --location ${{ env.REGION }} \
                --environment-size ${{ env.COMPOSER_ENV_SIZE }} \
                --image-version ${{ env.COMPOSER_IMAGE_VERSION }} \
                --service-account ${{ env.SERVICE_ACCOUNT_NAME }}@${{ secrets.PROJECT_ID }}.iam.gserviceaccount.com
        else
            echo "Composer environment ${{ env.COMPOSER_ENV_NAME }} already exists!"
        fi

    # Create composer environments
    - name: Create composer variable PROJECT_ID 
      run: |-
        gcloud composer environments run ${{ env.COMPOSER_ENV_NAME }} \
        --location ${{ env.REGION}} variables \
        -- set PROJECT_ID ${{ secrets.PROJECT_ID }}

    - name: Create composer variable REGION
      run: |-  
        gcloud composer environments run ${{ env.COMPOSER_ENV_NAME }} \
          --location ${{ env.REGION }} variables \
          -- set REGION ${{ env.REGION }}

    - name: Create composer variable CLOUD_FUNCTION_1_NAME
      run: |-
        gcloud composer environments run ${{ env.COMPOSER_ENV_NAME }}\
          --location ${{ env.REGION }} variables \
          -- set CLOUD_FUNCTION_1_NAME ${{ env.CLOUD_FUNCTION_1_NAME }}

    - name: Create composer variable CLOUD_FUNCTION_2_NAME
      run: |-
        gcloud composer environments run ${{ env.COMPOSER_ENV_NAME }} \
        --location ${{ env.REGION }} variables \
        -- set CLOUD_FUNCTION_2_NAME ${{ env.CLOUD_FUNCTION_2_NAME }}

    - name: Create composer variable CLOUD_FUNCTION_3_NAME
      run: |-
        gcloud composer environments run ${{ env.COMPOSER_ENV_NAME }} \
        --location ${{ env.REGION }} variables \
        -- set CLOUD_FUNCTION_3_NAME ${{ env.CLOUD_FUNCTION_3_NAME }}

    - name: Create composer variable BUCKET_DATALAKE
      run: |-
        gcloud composer environments run ${{ env.COMPOSER_ENV_NAME }} \
        --location ${{ env.REGION}} variables \
        -- set BUCKET_NAME ${{ secrets.BUCKET_DATALAKE }}

    - name: Create composer variable TRANSIENT_FILE_PATH
      run: |-
        gcloud composer environments run ${{ env.COMPOSER_ENV_NAME }} \
        --location ${{ env.REGION }} variables \
        -- set TRANSIENT_FILE_PATH ${{ env.TRANSIENT_FILE_PATH }}

    - name: Create composer variable BRONZE_PATH 
      run: |-
        gcloud composer environments run ${{ env.COMPOSER_ENV_NAME }} \
        --location ${{ env.REGION }} variables \
        -- set BRONZE_PATH ${{ env.BRONZE_PATH }}

    - name: Create composer variable SILVER_PATH
      run: |-
        gcloud composer environments run ${{ env.COMPOSER_ENV_NAME }} \
        --location ${{ env.REGION }} variables \
        -- set SILVER_PATH ${{ env.SILVER_PATH }}

    - name: Create composer variable REGION_PROJECT_ID
      run: |-
        gcloud composer environments run ${{ env.COMPOSER_ENV_NAME }} \
        --location ${{ env.REGION }} variables \
        -- set REGION_PROJECT_ID "${{ env.REGION }}-${{ secrets.PROJECT_ID }}"


    - name: Create composer variable BIGQUERY_DATASET
      run: |-
        gcloud composer environments run ${{ env.COMPOSER_ENV_NAME }} \
        --location ${{ env.REGION }} variables \
        -- set BIGQUERY_DATASET "${{ env.BIGQUERY_DATASET }}"

    - name: Create composer variable BIGQUERY_TABLE_CUSTOMER
      run: |-
        gcloud composer environments run ${{ env.COMPOSER_ENV_NAME }} \
        --location ${{ env.REGION }} variables \
        -- set BIGQUERY_TABLE_CUSTOMER "${{ env.BIGQUERY_TABLE_CUSTOMER }}"

Salin selepas log masuk

deploy-composer-http-connection

deploy-composer-http-connection:
    needs: [enable-services, deploy-buckets, deploy-cloud-function, deploy-composer-service-account, deploy-bigquery-dataset-bigquery-tables, deploy-composer-environment ]
    runs-on: ubuntu-22.04

    steps:
    - name: Checkout
      uses: actions/checkout@v4

    - name: Authorize GCP
      uses: 'google-github-actions/auth@v2'
      with:
        credentials_json:  ${{ secrets.GCP_DEVOPS_SA_KEY }}

    # Step to Authenticate with GCP
    - name: Set up Cloud SDK
      uses: google-github-actions/setup-gcloud@v2
      with:
        version: '>= 363.0.0'
        project_id: ${{ secrets.PROJECT_ID }}

    # Step to Configure Docker to use the gcloud command-line tool as a credential helper
    - name: Configure Docker
      run: |-
        gcloud auth configure-docker

    - name: Create composer http connection HTTP_CONNECTION
      run: |-
        HOST="https://${{ env.REGION }}-${{ secrets.PROJECT_ID }}.cloudfunctions.net"
        gcloud composer environments run ${{ env.COMPOSER_ENV_NAME }} \
        --location ${{ env.REGION }} connections \
        -- add ${{ env.HTTP_CONNECTION }} \
        --conn-type ${{ env.CONNECTION_TYPE  }} \
        --conn-host $HOST

Salin selepas log masuk

deploy-dags

deploy-dags:
    needs: [enable-services, deploy-buckets, deploy-cloud-function, deploy-composer-service-account, deploy-bigquery-dataset-bigquery-tables, deploy-composer-environment, deploy-composer-http-connection ]
    runs-on: ubuntu-22.04

    steps:
    - name: Checkout
      uses: actions/checkout@v4

    - name: Authorize GCP
      uses: 'google-github-actions/auth@v2'
      with:
        credentials_json:  ${{ secrets.GCP_DEVOPS_SA_KEY }}

    # Step to Authenticate with GCP
    - name: Set up Cloud SDK
      uses: google-github-actions/setup-gcloud@v2
      with:
        version: '>= 363.0.0'
        project_id: ${{ secrets.PROJECT_ID }}

    - name: Get Composer bucket name and Deploy DAG to Composer
      run: |-
        COMPOSER_BUCKET=$(gcloud composer environments describe ${{ env.COMPOSER_ENV_NAME }} \
        --location ${{ env.REGION }} \
        --format="value(config.dagGcsPrefix)")
        gsutil -m cp -r ./dags/* $COMPOSER_BUCKET/dags/

Salin selepas log masuk

Sumber Dicipta Selepas Penggunaan

Selepas proses penempatan selesai, sumber berikut akan tersedia:

Baldi Penyimpanan Awan: Disusun dalam lapisan Gangsa, Perak dan Emas.
Fungsi Cloud:: Bertanggungjawab untuk memproses data merentas tiga lapisan.
Akaun Perkhidmatan untuk Komposer:: Dengan kebenaran yang sesuai untuk menggunakan Fungsi Awan.
Set Data dan Jadual BigQuery: DataFrame dicipta untuk menyimpan data yang diproses.
Persekitaran Komposer Google: Mengatur Fungsi Awan dengan pelaksanaan harian.
DAG Komposer: DAG mengurus aliran kerja yang menggunakan Fungsi Awan dan memproses data.

Kesimpulan

Penyelesaian ini menunjukkan cara memanfaatkan Google Cloud Functions, Composer dan BigQuery untuk mencipta saluran paip pemprosesan data tiga peringkat yang mantap. Automasi menggunakan GitHub Actions memastikan proses penggunaan yang lancar dan boleh dihasilkan semula, menjadikannya lebih mudah untuk mengurus saluran paip data berasaskan awan pada skala.

Rujukan

Dokumentasi Platform Awan Google: https://cloud.google.com/docs
Dokumentasi Tindakan GitHub:: https://docs.github.com/en/actions
Dokumentasi Komposer Google:: https://cloud.google.com/composer/docs
Dokumentasi Cloud Functions: https://cloud.google.com/functions/docs
Repo GitHub: https://github.com/jader-lima/gcp-cloud-functions-to-bigquery

Atas ialah kandungan terperinci Menggunakan Fungsi Awan Google untuk Pemprosesan Data Tiga Peringkat dengan Komposer Google dan Penyerahan Automatik melalui Tindakan GitHub. Untuk maklumat lanjut, sila ikut artikel berkaitan lain di laman web China PHP!