- When viewing a config schema in the Dagit launchpad, default values are now shown. Hover over an underlined key in the schema view to see the default value for that key.
- dagster, dagit, and all extension libraries (dagster-*) now contain py.typed files. This exposes them as typed libraries to static type checking tools like mypy. If your project is using mypy or another type checker, this may surface new type errors. For mypy, to restore the previous state and treat dagster or an extension library as untyped (i.e. ignore Dagster’s type annotations), add the following to your configuration file:
[mypy-dagster] (or e.g. mypy-dagster-dbt)
follow_imports = "skip"
- Op retries now surface the underlying exception in Dagit.
- Made some internal changes to how we store schema migrations across our different storage implementations.
build_output_context now accepts an asset_key argument.- They key argument to the SourceAsset constructor now accepts values that are strings or sequences of strings and coerces them to AssetKeys.
- You can now use the + operator to add two AssetGroups together, which forms an AssetGroup that contains a union of the assets in the operands.
AssetGroup.from_package_module, from_modules, from_package_name, and from_current_module now accept an extra_source_assets argument that includes a set of source assets into the group in addition to the source assets scraped from modules.- AssetsDefinition and AssetGroup now both expose a
to_source_assets method that return SourceAsset versions of their assets, which can be used as source assets for downstream AssetGroups. - Repositories can now include multiple AssetGroups.
- The new prefixed method on AssetGroup returns a new AssetGroup where a given prefix is prepended to the asset key of every asset in the group.
- Dagster now has a BoolMetadataValue representing boolean-type metadata. Specifying True or False values in metadata will automatically be casted to the boolean type.
- Tags on schedules can now be expressed as nested JSON dictionaries, instead of requiring that all tag values are strings.
- If an exception is raised during an op, Dagster will now always run the failure hooks for that op. Before, certain system exceptions would prevent failure hooks from being run.
mapping_key can now be provided as an argument to build_op_context/build_solid_context. Doing so will allow the use of OpExecutionContext.get_mapping_key().
- [dagit] Previously, when viewing a list of an asset’s materializations from a specified date/time, a banner would always indicate that it was a historical view. This banner is no longer shown when viewing the most recent materialization.
- [dagit] Special cron strings like @daily were treated as invalid when converting to human-readable strings. These are now handled correctly.
- The selection argument to
AssetGroup.build_job now uses > instead of . for delimiting the components within asset keys, which is consistent with how selection works in Dagit. - [postgres] passwords and usernames are now correctly url quoted when forming a connection string. Previously spaces were replaced with
+. - Fixed an issue where the
celery_docker_executor would sometimes fail to execute with a JSON deserialization error when using Dagster resources that write to stdout. - [dagster-k8s] Fixed an issue where the Helm chart failed to work when the user code deployment subchart was used in a different namespace than the main dagster Helm chart, due to missing configmaps.
- [dagster-airbyte] When a Dagster run is terminated while executing an Airbyte sync operation, the corresponding Airbyte sync will also be terminated.
- [dagster-dbt] Log output from dbt cli commands will no longer have distracting color-formatting characters.
- [dagit] Fixed issue where multi_assets would not show correct asset dependency information.
- Fixed an issue with the sensor daemon, where the sensor would sometimes enter a race condition and overwrite the sensor status.
- [dagster-graphql] The Python DagsterGraphQLClient now supports terminating in-progress runs using
client.terminate_run(run_id). Thanks @Javier162380!
- Added an experimental view of the Partitions page / Backfill page, gated behind a feature flag in Dagit.
- [dagster-k8s] You can now specify resource requests and limits to the K8sRunLauncher when using the Dagster helm chart, that will apply to all runs. Before, you could only set resource configuration by tagging individual jobs. For example, you can set this config in your
values.yaml file:
runLauncher:
type: K8sRunLauncher
config:
k8sRunLauncher:
resources:
limits:
cpu: 100m
memory: 128Mi
requests:
cpu: 100m
memory: 128Mi
- [dagster-k8s] Specifying
includeConfigInLaunchedRuns: true in a user code deployment will now launch runs using the same namespace and service account as the user code deployment. - The
@asset decorator now accepts an op_tags argument, which allows e.g. providing k8s resource requirements on the op that computes the asset. - Added CLI output to
dagster api grpc-health-check (previously it just returned via exit codes) - [dagster-aws] The
emr_pyspark_step_launcher now supports dynamic orchestration, RetryPolicys defined on ops, and re-execution from failure. For failed steps, the stack trace of the root error will now be available in the event logs, as will logs generated with context.log.info. - Partition sets and can now return a nested dictionary in the
tags_fn_for_partition function, instead of requiring that the dictionary have string keys and values. - [dagit] It is now possible to perform bulk re-execution of runs from the Runs page. Failed runs can be re-executed from failure.
- [dagit] Table headers are now sticky on Runs and Assets lists.
- [dagit] Keyboard shortcuts may now be disabled from User Settings. This allows users with certain keyboard layouts (e.g. QWERTZ) to inadvertently avoid triggering unwanted shortcuts.
- [dagit] Dagit no longer continues making some queries in the background, improving performance when many browser tabs are open.
- [dagit] On the asset graph, you can now filter for multi-component asset keys in the search bar and see the “kind” tags displayed on assets with a specified compute_kind.
- [dagit] Repositories are now displayed in a stable order each time you launch Dagster.
- [dagster-k8s] Fixed an issue where the Dagster helm chart sometimes failed to parse container images with numeric tags. Thanks @jrouly!
- [dagster-aws] The
EcsRunLauncher now registers new task definitions if the task’s execution role or task role changes. - Dagster now correctly includes
setuptools as a runtime dependency. In can now accept asset_partitions without crashing.- [dagit] Fixed a bug in the Launchpad, where default configuration failed to load.
- [dagit] Global search now truncates the displayed list of results, which should improve rendering performance.
- [dagit] When entering an invalid search filter on Runs, the user will now see an appropriate error message instead of a spinner and an alert about a GraphQL error.
- Added documentation for partitioned assets
- [dagster-aws] Fixed example code of a job using
secretsmanager_resource.
- Fixed an issue where the Launchpad in Dagit sometimes incorrectly launched in an empty state.
- Fixed an issue where schedules created from partition sets that launched runs for multiple partitions in a single schedule tick would sometimes time out while generating runs in the scheduler.
- Fixed an issue where nested graphs would sometimes incorrectly determine the set of required resources for a hook.
- [dagster-k8s] Added an
includeConfigInLaunchedRuns flag to the Helm chart that can be used to automatically include configmaps, secrets, and volumes in any runs launched from code in a user code deployment. See https://docs.dagster.io/deployment/guides/kubernetes/deploying-with-helm#configure-your-user-deployment for more information. - [dagit] Improved display of configuration yaml throughout Dagit, including better syntax highlighting and the addition of line numbers.
- The GraphQL input argument type
BackfillParams (used for launching backfills), now has an allPartitions boolean flag, which can be used instead of specifying all the individual partition names. - Removed
gevent and gevent-websocket dependencies from dagster-graphql - Memoization is now supported while using step selection
- Cleaned up various warnings across the project
- The default IO Managers now support asset partitions
Fixed sqlite3.OperationalError error when viewing schedules/sensors pages in Dagit. This was affecting dagit instances using the default SQLite schedule storage with a SQLite version < 3.25.0.
Fixed an issues where schedules and sensors would sometimes fail to run when the daemon and dagit were running in different Python environments.
Fixed an exception when the telemetry file is empty
fixed a bug with @graph composition which would cause the wrong input definition to be used for type checks
[dagit] For users running Dagit with --path-prefix, large DAGs failed to render due to a WebWorker error, and the user would see an endless spinner instead. This has been fixed.
[dagit] Fixed a rendering bug in partition set selector dropdown on Launchpad.
[dagit] Fixed the ‘View Assets’ link in Job headers
Fixed an issue where root input managers with resource dependencies would not work with software defined assets
dagster-census is a new library that includes a census_resource for interacting the Census REST API, census_trigger_sync_op for triggering a sync and registering an asset once it has finished, and a CensusOutput type. Thanks @dehume!- Docs fix. Thanks @ascrookes!