Set up the Atlassian Jira Cloud (Core) flow¶

Prerequisites¶

1. Ensure that you have reviewed About Openflow Connector for Jira Cloud.

2. Ensure that you have Set up Openflow - BYOC or Set up Openflow - Snowflake Deployments.

3. If using Openflow - Snowflake Deployments, ensure that you’ve reviewed configuring required domains and have granted access to the required domains for the Jira Cloud connector.

Get the credentials¶

As a Jira Cloud administrator, perform the following tasks in your Atlassian account:

1. Navigate to the API tokens page.

2. Select Create API token with scopes.

3. In the Create an API token dialog box, provide a descriptive name for the API token and select an expiration date for the API token. This can range from 1 to 365 days.

4. Select the API token app Jira.

5. Select the required scopes based on the features you plan to use. See Required API scopes for details.

6. Select Create token.

7. In the Copy your API token dialog box, select Copy to copy your generated API token and then paste the token to the connector parameters, or save it securely.

8. Select Close to close the dialog box.

Set up Snowflake account¶

As a Snowflake account administrator, perform the following tasks:

1. Create a new role or use an existing role.

2. Create a new Snowflake service user with the type as SERVICE.

3. Grant the Snowflake service user the role you created in the previous steps.

4. Configure with key-pair auth for the Snowflake SERVICE user from step 2.

5. Configure a secrets manager supported by Openflow (recommended), for example, AWS, Azure, and HashiCorp, and store the public and private keys in the secret store.

   Note

   If for any reason, you don’t want to use a secrets manager, then you are responsible for safeguarding the public key and private key files used for key-pair authentication according to the security policies of your organization.

   1. After the secrets manager is configured, determine how you will authenticate to it. On AWS, use the EC2 instance role associated with Openflow as this way no other secrets have to be persisted.

   2. In Openflow, configure a Parameter Provider associated with this Secrets Manager, from the main menu (⋮) in the upper-right corner. Navigate to Controller Settings » Parameter Provider and then fetch your parameter values.

   3. At this point, all credentials can be referenced with the associated parameter paths and no sensitive values need to be persisted within Openflow.

6. If any other Snowflake users require access to the tables ingested by the connector (for example, for custom processing in Snowflake), then grant those users the role created in step 1.

7. Create a database and schema in Snowflake for the connector to store ingested data. Grant the following Database privileges to the role created in the first step.

   CopyExpand

   CREATE DATABASE jira_destination_db;
   CREATE SCHEMA jira_destination_db.jira_destination_schema;
   GRANT USAGE ON DATABASE jira_destination_db TO ROLE <jira_connector_role>;
   GRANT USAGE ON SCHEMA jira_destination_db.jira_destination_schema TO ROLE <jira_connector_role>;
   GRANT CREATE TABLE ON SCHEMA jira_destination_db.jira_destination_schema TO ROLE <jira_connector_role>;
   

   Show lessSee more

   Scroll to top

8. Create a warehouse that the connector will use or use an existing one. Start with the smallest warehouse size, then experiment with size depending on the amount of data transferred. Large data volumes typically scale better with multi-cluster warehouses, rather than larger warehouse sizes.

9. Ensure that the user with the role used by the connector has the required privileges to use the warehouse. If that’s not the case then grant the required privileges to the role.

   CopyExpand

   CREATE WAREHOUSE jira_connector_warehouse WITH WAREHOUSE_SIZE = 'X-Small';
   GRANT USAGE ON WAREHOUSE jira_connector_warehouse TO ROLE <jira_connector_role>;
   

   Show lessSee more

   Scroll to top

Set up the connector¶

The core flow is shipped as the Atlassian Jira Cloud (Core) process group. As a data engineer, perform the following tasks to install and configure it.

Run the flow¶

1. Right-click on the canvas and select Enable all Controller Services.

2. Right-click on the Atlassian Jira Cloud (Core) process group and select Start. The flow starts the data ingestion.

On first run, the flow creates the required Snowflake tables in the destination schema. See Destination tables for the full list of tables created by the core flow and the parameters that control which optional tables are populated.

Resetting the connector state¶

If you need to change the project filter or want to restart the ingestion from scratch, you must clear the ingestion state. The core flow uses a centralized state service rather than per-processor state.

To reset the state, perform the following steps:

1. Right-click the Atlassian Jira Cloud (Core) process group and select Stop.

2. Navigate to the Controller Settings for the process group.

3. Find the StandardJiraIngestionStateService controller service and select View State.

4. Select Clear State. This clears all project tracking, pagination, and timestamp state.

5. Optionally, update the connector parameters if needed.

6. Right-click the Atlassian Jira Cloud (Core) process group and select Start.

Accessing the data¶

Data fetched from Jira is available in the destination tables with explicit column schemas. There is no need to use JSON flattening or views to query the data.

Each entity is stored in its own table. For example, to query issues and their comments:

SELECT i.KEY, i.SUMMARY, c.BODY AS comment_body, c.CREATED AS comment_created
FROM ISSUE i
JOIN COMMENT c ON i.ID = c.ISSUE_ID
ORDER BY c.CREATED DESC;

To exclude deleted issues from query results, filter on the connector-managed _SNOWFLAKE_DELETED column. The connector sets this flag to TRUE on the matching ISSUE row when an issue is deleted in Jira, so no anti-join against DELETED_ISSUE is needed:

SELECT i.*
FROM ISSUE i
WHERE i._SNOWFLAKE_DELETED = FALSE;

The DELETED_ISSUE table is still useful when you need the deletion timestamp or the user who performed the deletion. See Connector-managed columns for the full set of connector-managed metadata columns.

Enabled tables configuration¶

The Enabled Tables parameter controls which optional tables are populated. Ingestion of the ISSUE, PROJECT, USER, and FIELD tables is always enabled and can’t be disabled. Enabling all tables may cause performance issues and require a larger runtime.

Available values for Enabled Tables:

• CHANGELOG (field change history for issues)

• COMMENT (comments on issues)

• ISSUE_REMOTE_LINK (remote links attached to issues)

• ISSUE_SECURITY_SCHEME (issue-level security configurations)

• ISSUE_VOTE (users who voted on issues)

• ISSUE_WATCHER (users watching issues)

• PERMISSION (global and project permission definitions)

• PROJECT_COMPONENT (components defined in a project)

• PROJECT_VERSION (release versions of a project)

• USER_GROUP (group memberships per user)

• WORKLOG (time tracking entries on issues)

The per-issue tables (CHANGELOG, COMMENT, ISSUE_REMOTE_LINK, ISSUE_VOTE, ISSUE_WATCHER, WORKLOG) and per-project tables (PROJECT_COMPONENT, PROJECT_VERSION) only ingest data for issues and projects that are also covered by Project Keys Filter.

Some tables are populated by calling the Jira API once per parent entity (for example, once per user or once per issue). On large Jira instances, enabling these tables can significantly increase the number of API calls and the load on the ingestion runtime, and can slow down population of the parent table due to back-pressure on the upstream processor. Enable these tables only when the corresponding data is required.

Issue fields configuration¶

The ISSUE table schema depends on the Issue Fields parameter. The parameter accepts a comma-separated list of field IDs or one of the special values below. Prefix a field with a minus (-) to exclude it. For example, *all,-description returns all fields except description.

• *standard (default): Fetches standard Jira fields (Summary, Status, Priority, Assignee).

• *navigable: Fetches all navigable fields.

• *all: Fetches all fields, including custom fields.

• Individual field IDs can be specified (for example, summary,status,customfield_10001).

The default value *standard doesn’t include custom fields. To ingest custom fields, set this parameter to *all or list the custom fields explicitly by ID, for example, *standard,customfield_10001. To find custom field IDs, follow this guide.

Column names in the ISSUE table are derived from Jira field display names by:

1. Uppercasing the display name.

2. Replacing spaces with underscores.

3. Removing every character that isn’t a letter, digit, or underscore.

For example, the display name OF Test (Multi-User) becomes the column OF_TEST_MULTIUSER.

If two fields produce the same column name after this transformation, the second field’s column is suffixed with __<flattened_field_id> to keep names unique. For example, two fields with display name Custom Field and IDs customfield_1 and customfield_2 produce columns CUSTOM_FIELD and CUSTOM_FIELD__CUSTOMFIELD_2.

Jira field types are mapped to Snowflake column types as follows:

Next steps¶

• Set up the Atlassian Jira Cloud (Agile) flow to install the agile flow.

• Migrate from the legacy Openflow Connector for Jira Cloud if you’re moving from a previous version of the Jira Cloud connector.