Skip to main content

GitHub (dagster-github)

This library provides an integration with GitHub Apps, to support performing various automation operations within your github repositories and with the tighter permissions scopes that github apps allow for vs using a personal token.

Presently, it provides a thin wrapper on the github v4 graphql API.

To use this integration, you’ll first need to create a GitHub App for it.

  1. Create App: Follow the instructions in https://developer.github.com/apps/quickstart-guides/setting-up-your-development-environment/, You will end up with a private key and App ID, which will be used when configuring the dagster-github resource. Note you will need to grant your app the relevent permissions for the API requests you want to make, for example to post issues it will need read/write access for the issues repository permission, more info on GitHub application permissions can be found here
  2. Install App: Follow the instructions in https://developer.github.com/apps/quickstart-guides/setting-up-your-development-environment/#step-7-install-the-app-on-your-account
  3. Find your installation_id: You can pull this from the GitHub app administration page, https://github.com/apps/<app-name>/installations/<installation_id>. Note if your app is installed more than once you can also programatically retrieve these IDs. Sharing your App ID and Installation ID is fine, but make sure that the Private Key for your app is stored securily.

Posting Issues

Now, you can create issues in GitHub from Dagster with the GitHub resource:

import os

from dagster import job, op
from dagster_github import GithubResource


@op
def github_op(github: GithubResource):
    github.get_client().create_issue(
        repo_name='dagster',
        repo_owner='dagster-io',
        title='Dagster\'s first github issue',
        body='this open source thing seems like a pretty good idea',
    )

@job(resource_defs=\{
     'github': GithubResource(
         github_app_id=os.getenv('GITHUB_APP_ID'),
         github_app_private_rsa_key=os.getenv('GITHUB_PRIVATE_KEY'),
         github_installation_id=os.getenv('GITHUB_INSTALLATION_ID')
 )})
def github_job():
    github_op()

github_job.execute_in_process()

Run the above code, and you’ll see the issue appear in GitHub:

GitHub enterprise users can provide their hostname in the run config. Provide github_hostname as part of your github config like below.

GithubResource(
    github_app_id=os.getenv('GITHUB_APP_ID'),
    github_app_private_rsa_key=os.getenv('GITHUB_PRIVATE_KEY'),
    github_installation_id=os.getenv('GITHUB_INSTALLATION_ID'),
    github_hostname=os.getenv('GITHUB_HOSTNAME'),
)

By provisioning GithubResource as a Dagster resource, you can post to GitHub from within any asset or op execution.

Executing GraphQL queries

import os

from dagster import job, op
from dagster_github import github_resource


@op
def github_op(github: GithubResource):
    github.get_client().execute(
        query="""
        query get_repo_id($repo_name: String!, $repo_owner: String!) \{
            repository(name: $repo_name, owner: $repo_owner) \{
                id
            }
        }
        """,
        variables=\{"repo_name": repo_name, "repo_owner": repo_owner},
    )

@job(resource_defs=\{
     'github': GithubResource(
         github_app_id=os.getenv('GITHUB_APP_ID'),
         github_app_private_rsa_key=os.getenv('GITHUB_PRIVATE_KEY'),
         github_installation_id=os.getenv('GITHUB_INSTALLATION_ID')
 )})
def github_job():
    github_op()

github_job.execute_in_process()
dagster_github.GithubResource ResourceDefinition

Base class for Dagster resources that utilize structured config.

This class is a subclass of both ResourceDefinition and Config.

Example definition:

class WriterResource(ConfigurableResource):
    prefix: str

    def output(self, text: str) -> None:
        print(f"\{self.prefix}\{text}")

Example usage:

@asset
def asset_that_uses_writer(writer: WriterResource):
    writer.output("text")

defs = Definitions(
    assets=[asset_that_uses_writer],
    resources=\{"writer": WriterResource(prefix="a_prefix")},
)

You can optionally use this class to model configuration only and vend an object of a different type for use at runtime. This is useful for those who wish to have a separate object that manages configuration and a separate object at runtime. Or where you want to directly use a third-party class that you do not control.

To do this you override the create_resource methods to return a different object.

class WriterResource(ConfigurableResource):
    str: prefix

    def create_resource(self, context: InitResourceContext) -> Writer:
        # Writer is pre-existing class defined else
        return Writer(self.prefix)

Example usage:

@asset
def use_preexisting_writer_as_resource(writer: ResourceParam[Writer]):
    writer.output("text")

defs = Definitions(
    assets=[use_preexisting_writer_as_resource],
    resources=\{"writer": WriterResource(prefix="a_prefix")},
)