使用声明式自动化捆绑包管理功能视图

Important

此功能以个人预览版提供。 若要试用,请联系 Azure Databricks 联系人。

声明式自动化捆绑包(前称 Databricks 资产捆绑包)允许您通过程序验证、部署并运行 Azure Databricks 资源,如功能视图及其实现的流水线。 请参阅什么是声明性自动化捆绑包?

本教程构建了一个捆绑包,声明了四个 功能视图,并以三种方式将其物化:两个批量特征同时写入离线和在线存储、一个持续写入在线存储的流式特征,以及一个带锯齿形窗口的流式特征,该特征在批处理模式下物化到离线和在线存储,并在流式模式下物化到在线存储。 然后你修改配置,重新部署,然后清理所有资源。

在捆绑包中声明特征可确保在你部署到的每个工作区中都使用相同的定义,这些定义将与项目的其他内容一起作为源文件进行审查。 它是从笔记本中调用 materialize_features 和 create_feature 的声明式替代方案。 请参阅 功能视图。

每一步都显示了YAML和Python的配置。 选择一种语言,页面会跟随你的选择。

Requirements

  • Databricks CLI 版本 1.24.0 或更高。 运行 databricks --version 以检查。 请参阅安装或更新 Databricks CLI。
  • 直接部署引擎。 features 和 materialization_sets 资源不受 Terraform 部署引擎支持。 参见 资源兼容性。
  • 启用 Unity Catalog 的工作区。 请参阅 要求。
  • 创建并具体化特征的权限。 请参阅 权限。
  • 一个现有的在线商店。 参见 创建在线商店。
  • 一个现有的Delta表,包含实体列和时间戳列,用于批量物化。 请参阅 Azure Databricks 表。
  • 一个现有流,用于流具体化。 参见 创建流。

源表和流各自有自己的生命周期,且都不是捆绑资源。 先创建它们,并在捆绑包中用名称引用,就像 步骤3 对表格和 第4步 对流一样。

捆绑中的 Feature Store 资源

本教程声明两种 Feature Store 资源类型,它们之间的区别决定部署的成本。

  • 特征视图:特征的定义,涵盖其来源、聚合和键。 它是轻量级元数据。 请参阅特征视图;有关配置参考,请参阅特征。
  • 实体化集:一组 实体化特征。 实体化特征是通过管道或作业计算到表格中的一个特征视图,因此这是重量级的部分。 将实体化特征归为一组,告诉 Azure Databricks 可以将它们合并计算,因此它们被压缩到包含这些特征的最少表和 Lakeflow 管道中。 参见 Materialize特征视图,以及配置参考, materialization_set 和 materialization_set.materialized_features。

接下来的步骤包括声明批处理特征、流式特征和带有锯齿窗口的流式特征。 只声明你的项目需要的。 关于具体化的工作原理,请参阅具体化特征视图。

步骤 1:设置身份验证

首先,在你的开发机器上的Databricks CLI和你的Azure Databricks工作区之间设置身份验证。 此页假定你想要使用 OAuth 用户到机器(U2M)身份验证,以及一个名为 DEFAULT 的相应 Azure Databricks 配置文件进行身份验证。

Note

U2M 身份验证适用于实时尝试这些步骤。 对于完全自动化的工作流,Databricks 建议改用 OAuth 计算机到计算机 (M2M) 身份验证。 请参阅 授权服务主体使用 OAuth 访问 Azure Databricks 中的 M2M 身份验证设置说明。

  1. 通过对每个目标工作区运行以下命令,使用 Databricks CLI 在本地启动 OAuth 令牌管理。

    在以下命令中,将 <workspace-url> 替换为 Azure Databricks per-workspace URL,例如 https://adb-1234567890123456.7.azuredatabricks.net。

    databricks auth login --host <workspace-url>
    
  2. Databricks CLI 会提示您将输入的信息另存为 Azure Databricks 配置文件。 按 Enter 确认建议的配置文件名称,或输入新配置文件或已有配置文件的名称。 使用输入的信息覆盖同名的任何现有配置文件。 可以使用个人资料在多个工作区之间快速切换身份验证上下文。

    若要获取任何现有配置文件的列表,请在单独的终端或命令提示符中使用 Databricks CLI 来运行 databricks auth profiles 命令。 要查看特定配置文件的现有设置,请运行命令 databricks auth env --profile <profile-name>。

  3. 在 Web 浏览器中,按照屏幕上的说明登录到 Azure Databricks 工作区。

  4. 要查看个人资料的当前 OAuth 令牌值和令牌即将到期的时间戳,请运行以下命令之一:

    • databricks auth token --host <workspace-url>
    • databricks auth token -p <profile-name>
    • databricks auth token --host <workspace-url> -p <profile-name>

    如果你有多个配置文件具有相同的--host 值,则可能会需要同时指定 --host 和 -p 选项,以帮助 Databricks CLI 找到正确的匹配的 OAuth 令牌信息。

步骤 2:创建捆绑包

为捆绑包创建一个空目录,根目录 databricks.yml 有一个文件,旁边有一个 resources 目录。

YAML

以下每一步都会在 resources/ 下增加一个文件,用于声明捆集资源:

databricks.yml
resources/
  batch_feature.yml
  streaming_feature.yml
  streaming_feature_sawtooth.yml

databricks.yml 有三点:

  • 包的名称。
  • 该 include 条目决定资源包从哪个文件加载资源。
  • 这些变量保存了资源可以引用的值,如目录、模式和源名称。
bundle:
  name: feature-store-quickstart

include:
  - resources/*.yml

variables:
  catalog:
    description: The catalog that features are written to.
    default: main
  schema:
    description: The schema that features are written to.
    default: features_demo
  source_table:
    description: The three-level name of the source Delta table.
    default: main.features_demo.transactions
  stream:
    description: The three-level name of a Stream registered in Unity Catalog.
    default: main.features_demo.events
  online_store:
    description: The online store that serves the features.
    default: my_online_store

把所有东西都声明进 databricks.yml 去也行。 省略 include 并收集每步的声明,集中在一个 resources 映射下:

bundle:
  name: feature-store-quickstart

variables:
  # The variables above.

resources:
  features:
    # The Feature Views from the following steps.
  materialization_sets:
    # The materialization sets from the following steps.

Python

Python 中声明的资源只能通过databricks.yml中的python映射找到。 没有它,捆绑包可以成功验证并部署,但没有资源,所以现在就添加它:

bundle:
  name: feature-store-quickstart

python:
  venv_path: .venv
  resources:
    - 'resources:load_resources'

variables:
  catalog:
    description: The catalog that features are written to.
    default: main
  schema:
    description: The schema that features are written to.
    default: features_demo
  source_table:
    description: The three-level name of the source Delta table.
    default: main.features_demo.transactions
  stream:
    description: The three-level name of a Stream registered in Unity Catalog.
    default: main.features_demo.events
  online_store:
    description: The online store that serves the features.
    default: my_online_store

venv_path 是Databricks CLI运行这些代码的虚拟环境。 resources 下的每个条目均以 <module>:<function> 形式指定一个函数,Databricks CLI 会调用该函数来收集捆绑包的资源。 如果在 <module>:<function> 下声明了多个 resources 条目,Databricks CLI 会调用每个函数并收集每个函数返回的资源。

然后创建项目及其虚拟环境,添加以下包:

uv init --bare
uv venv .venv
uv add databricks-bundles

创建 resources/__init__.py,其中包含 databricks.yml 指定的函数:

from databricks.bundles.core import Bundle, Resources


def load_resources(bundle: Bundle) -> Resources:
    resources = Resources()
    # The following steps add resources here.
    return resources

关于使用 Python 编写捆绑包的更多信息,包括项目模板和变更器,请参见 捆绑包配置(Python)。

步骤3:添加批处理特征

这两个 Feature View 会在过去七天和过去 30 天内为每位用户统计交易,均每日重新计算,并共同物化到离线和在线存储中。

单独的特征视图只是一个定义。 具体化集是从中生成数据的对象。

每种资源都有两个名称。 transaction_count 是 资源键:你选择的名称,在捆包中唯一,供 databricks bundle 命令用来指向该资源,并供其他配置引用。 例如,集合${resources.features.transaction_count.full_name}中的两个transaction_count实体化特征将特征视图的名称引用为transaction_count,这告诉 bundle 该集合依赖于特征视图transactions。

由于这种依赖关系,当 transaction_count 两者被创建时,先于集合 transactions 创建,在两者被摧毁后被删除。 像 main.features_demo.transaction_count_7d 这样的硬编码名称,如果不写 ${},则省略了该依赖关系,部署顺序将不可预测。

full_name 是你为 Feature View 选择的、在 Unity Catalog 下注册的三级名称,也是你之后查询时使用的名称。 这两个名字是独立的。

持续时间是指秒数加上 s 后缀,比如 604800s 表示七天。 参见 “指定窗口、源或函数的持续时间”。

在 materialized_features 下的每个条目都是一个已物化的特征成员:一个特征视图物化到一个目的地。 每个特征视图同时具现于离线和在线目的地,因此每个节点贡献两个具象化的特征成员,每个成员都有自己的触发条件。 两个功能视图写入同一个离线和在线目的地,因为一组只有一个离线目的地,最多一个在线目标。

YAML

创建 resources/batch_feature.yml:

resources:
  features:
    transaction_count:
      full_name: ${var.catalog}.${var.schema}.transaction_count_7d
      description: Transactions per user over a trailing seven days
      source:
        delta_table_source:
          full_name: ${var.source_table}
      function:
        aggregation_function:
          count_function:
            input: transaction_id
          time_window:
            sliding:
              window_duration: 604800s
              slide_duration: 86400s
      entities:
        - name: user_id
      timeseries_column:
        name: transaction_time

    transaction_count_30d:
      full_name: ${var.catalog}.${var.schema}.transaction_count_30d
      description: Transactions per user over a trailing 30 days
      source:
        delta_table_source:
          full_name: ${var.source_table}
      function:
        aggregation_function:
          count_function:
            input: transaction_id
          time_window:
            sliding:
              window_duration: 2592000s
              slide_duration: 86400s
      entities:
        - name: user_id
      timeseries_column:
        name: transaction_time

  materialization_sets:
    transactions:
      materialized_features:
        # Batch offline materialized feature of transaction_count.
        - feature_name: ${resources.features.transaction_count.full_name}
          offline_store_config:
            catalog_name: ${var.catalog}
            schema_name: ${var.schema}
            table_name_prefix: transactions_offline
          cron_schedule_trigger:
            mode: DERIVED
        # Batch online materialized feature of transaction_count.
        - feature_name: ${resources.features.transaction_count.full_name}
          online_store_config:
            online_store_name: ${var.online_store}
            catalog_name: ${var.catalog}
            schema_name: ${var.schema}
            table_name_prefix: transactions_online
          cron_schedule_trigger:
            mode: DERIVED
        # Batch offline materialized feature of transaction_count_30d.
        - feature_name: ${resources.features.transaction_count_30d.full_name}
          offline_store_config:
            catalog_name: ${var.catalog}
            schema_name: ${var.schema}
            table_name_prefix: transactions_offline
          cron_schedule_trigger:
            mode: DERIVED
        # Batch online materialized feature of transaction_count_30d.
        - feature_name: ${resources.features.transaction_count_30d.full_name}
          online_store_config:
            online_store_name: ${var.online_store}
            catalog_name: ${var.catalog}
            schema_name: ${var.schema}
            table_name_prefix: transactions_online
          cron_schedule_trigger:
            mode: DERIVED

Python

将导入添加到 resources/__init__.py:

from databricks.bundles.features import (
    AggregationFunction,
    CountFunction,
    DataSource,
    DeltaTableSource,
    EntityColumn,
    Feature,
    Function,
    SlidingWindow,
    TimeWindow,
    TimeseriesColumn,
)
from databricks.bundles.materialization_sets import (
    CronSchedule,
    CronScheduleMode,
    MaterializationSet,
    MaterializedFeature,
    OfflineStoreConfig,
    OnlineStoreConfig,
)

然后将此加到正 load_resources文中:

resources.add_feature(
    'transaction_count',
    Feature(
        full_name='${var.catalog}.${var.schema}.transaction_count_7d',
        description='Transactions per user over a trailing seven days',
        source=DataSource(delta_table_source=DeltaTableSource(full_name='${var.source_table}')),
        function=Function(
            aggregation_function=AggregationFunction(
                count_function=CountFunction(input='transaction_id'),
                time_window=TimeWindow(
                    sliding=SlidingWindow(window_duration='604800s', slide_duration='86400s')
                ),
            )
        ),
        entities=[EntityColumn(name='user_id')],
        timeseries_column=TimeseriesColumn(name='transaction_time'),
    ),
)

resources.add_feature(
    'transaction_count_30d',
    Feature(
        full_name='${var.catalog}.${var.schema}.transaction_count_30d',
        description='Transactions per user over a trailing 30 days',
        source=DataSource(delta_table_source=DeltaTableSource(full_name='${var.source_table}')),
        function=Function(
            aggregation_function=AggregationFunction(
                count_function=CountFunction(input='transaction_id'),
                time_window=TimeWindow(
                    sliding=SlidingWindow(window_duration='2592000s', slide_duration='86400s')
                ),
            )
        ),
        entities=[EntityColumn(name='user_id')],
        timeseries_column=TimeseriesColumn(name='transaction_time'),
    ),
)

resources.add_materialization_set(
    'transactions',
    MaterializationSet(
        materialized_features=[
            # Batch offline materialized feature of transaction_count.
            MaterializedFeature(
                feature_name='${resources.features.transaction_count.full_name}',
                offline_store_config=OfflineStoreConfig(
                    catalog_name='${var.catalog}',
                    schema_name='${var.schema}',
                    table_name_prefix='transactions_offline',
                ),
                cron_schedule_trigger=CronSchedule(mode=CronScheduleMode.DERIVED),
            ),
            # Batch online materialized feature of transaction_count.
            MaterializedFeature(
                feature_name='${resources.features.transaction_count.full_name}',
                online_store_config=OnlineStoreConfig(
                    online_store_name='${var.online_store}',
                    catalog_name='${var.catalog}',
                    schema_name='${var.schema}',
                    table_name_prefix='transactions_online',
                ),
                cron_schedule_trigger=CronSchedule(mode=CronScheduleMode.DERIVED),
            ),
            # Batch offline materialized feature of transaction_count_30d.
            MaterializedFeature(
                feature_name='${resources.features.transaction_count_30d.full_name}',
                offline_store_config=OfflineStoreConfig(
                    catalog_name='${var.catalog}',
                    schema_name='${var.schema}',
                    table_name_prefix='transactions_offline',
                ),
                cron_schedule_trigger=CronSchedule(mode=CronScheduleMode.DERIVED),
            ),
            # Batch online materialized feature of transaction_count_30d.
            MaterializedFeature(
                feature_name='${resources.features.transaction_count_30d.full_name}',
                online_store_config=OnlineStoreConfig(
                    online_store_name='${var.online_store}',
                    catalog_name='${var.catalog}',
                    schema_name='${var.schema}',
                    table_name_prefix='transactions_online',
                ),
                cron_schedule_trigger=CronSchedule(mode=CronScheduleMode.DERIVED),
            ),
        ]
    ),
)

步骤4:添加流式特征

该功能视图会在过去90分钟内统计每个键的事件,并持续物化到在线存储。 它写入一个目的地,因此其集合声明单个成员。

流式具体化特征成员携带streaming_mode,且不携带cron_schedule_trigger。 关于模式及其含义,请参见 materialization_set.materialized_features.streaming_mode。

这是第二个实体化集,而不是添加到 第 3 步所定义集合中的实体化特征,因为实体化集不能将以流式模式实体化的特征视图与以批处理方式实体化的特征视图混合使用。 带有锯齿窗的流式 Feature View 是例外。 它的批量和流式物化特征属于同一组,就像 第5步一样。

功能视图按名称读取流。 流是独立创建并进行版本管理的,在捆包之外,方式与批处理功能读取的 Delta 表相同。 要创建一个流,请参见 创建流。

流式特征视图将输入列作为流记录中的路径来引用,而非普通列名,因此在示例显示 value.* 的位置,请将示例中的名称替换为您的 Stream 的架构所使用的名称。

YAML

创建 resources/streaming_feature.yml:

resources:
  features:
    event_count_90m:
      full_name: ${var.catalog}.${var.schema}.event_count_90m
      description: Events per key over a trailing 90 minutes
      source:
        stream_source:
          full_name: ${var.stream}
      function:
        aggregation_function:
          count_function:
            input: value.amount
          time_window:
            rolling:
              window_duration: 5400s
      entities:
        - name: value.key
      timeseries_column:
        name: value.event_timestamp

  materialization_sets:
    events_online:
      materialized_features:
        - feature_name: ${resources.features.event_count_90m.full_name}
          online_store_config:
            online_store_name: ${var.online_store}
            catalog_name: ${var.catalog}
            schema_name: ${var.schema}
            table_name_prefix: event_count_90m_online
          streaming_mode:
            mode: STREAMING_MODE_TYPE_RTM

Python

将 StreamSource 和 RollingWindow 添加到从 databricks.bundles.features 导入的内容中,并将 StreamingMode 和 StreamingModeStreamingModeType 添加到从 databricks.bundles.materialization_sets 导入的内容中,然后将以下内容添加到 load_resources:

resources.add_feature(
    'event_count_90m',
    Feature(
        full_name='${var.catalog}.${var.schema}.event_count_90m',
        description='Events per key over a trailing 90 minutes',
        source=DataSource(stream_source=StreamSource(full_name='${var.stream}')),
        function=Function(
            aggregation_function=AggregationFunction(
                count_function=CountFunction(input='value.amount'),
                time_window=TimeWindow(rolling=RollingWindow(window_duration='5400s')),
            )
        ),
        entities=[EntityColumn(name='value.key')],
        timeseries_column=TimeseriesColumn(name='value.event_timestamp'),
    ),
)

resources.add_materialization_set(
    'events_online',
    MaterializationSet(
        materialized_features=[
            MaterializedFeature(
                feature_name='${resources.features.event_count_90m.full_name}',
                online_store_config=OnlineStoreConfig(
                    online_store_name='${var.online_store}',
                    catalog_name='${var.catalog}',
                    schema_name='${var.schema}',
                    table_name_prefix='event_count_90m_online',
                ),
                streaming_mode=StreamingMode(
                    mode=StreamingModeStreamingModeType.STREAMING_MODE_TYPE_RTM
                ),
            )
        ]
    ),
)

步骤5:添加带有锯齿形窗口的流式特征

对使用锯齿窗口进行聚合的流式特征视图进行实体化,会生成三个实体化特征:离线批量实体化特征、在线批量实体化特征和在线流式实体化特征。 服务器要求这三者全部,且声明部分具象化特征的子集会被拒绝。 这两个批量具体化特征必须将 DERIVED 作为 cron_schedule_trigger 使用,这样 Azure Databricks 才能从特征定义中选择调度。 批量在线实体化特征和流式在线实体化特征必须共享相同的在线商店配置。 不满足这些要求的实体化集在部署时被拒绝。 完整列表请参见 “物质化集合必须满足的规则”。

YAML

创建 resources/streaming_feature_sawtooth.yml:

resources:
  features:
    amount_sum:
      full_name: ${var.catalog}.${var.schema}.amount_sum_3d
      description: Sum of amount per key over a sawtooth window
      source:
        stream_source:
          full_name: ${var.stream}
      function:
        aggregation_function:
          sum:
            input: value.amount
          time_window:
            sawtooth:
              window_duration: 259200s
      entities:
        - name: value.key
      timeseries_column:
        name: value.event_timestamp

  materialization_sets:
    amounts:
      materialized_features:
        # Batch offline materialized feature.
        - feature_name: ${resources.features.amount_sum.full_name}
          offline_store_config:
            catalog_name: ${var.catalog}
            schema_name: ${var.schema}
            table_name_prefix: amount_sum_offline
          cron_schedule_trigger:
            mode: DERIVED
        # Batch online materialized feature.
        - feature_name: ${resources.features.amount_sum.full_name}
          online_store_config:
            online_store_name: ${var.online_store}
            catalog_name: ${var.catalog}
            schema_name: ${var.schema}
            table_name_prefix: amount_sum_online
          cron_schedule_trigger:
            mode: DERIVED
        # Streaming online materialized feature.
        - feature_name: ${resources.features.amount_sum.full_name}
          online_store_config:
            online_store_name: ${var.online_store}
            catalog_name: ${var.catalog}
            schema_name: ${var.schema}
            table_name_prefix: amount_sum_online
          streaming_mode:
            mode: STREAMING_MODE_TYPE_RTM

Python

将 databricks.bundles.features 和 SumFunction 添加到从 SawtoothWindow 导入的内容中,然后将此添加到 load_resources:

resources.add_feature(
    'amount_sum',
    Feature(
        full_name='${var.catalog}.${var.schema}.amount_sum_3d',
        description='Sum of amount per key over a sawtooth window',
        source=DataSource(stream_source=StreamSource(full_name='${var.stream}')),
        function=Function(
            aggregation_function=AggregationFunction(
                sum=SumFunction(input='value.amount'),
                time_window=TimeWindow(sawtooth=SawtoothWindow(window_duration='259200s')),
            )
        ),
        entities=[EntityColumn(name='value.key')],
        timeseries_column=TimeseriesColumn(name='value.event_timestamp'),
    ),
)

resources.add_materialization_set(
    'amounts',
    MaterializationSet(
        materialized_features=[
            # Batch offline materialized feature.
            MaterializedFeature(
                feature_name='${resources.features.amount_sum.full_name}',
                offline_store_config=OfflineStoreConfig(
                    catalog_name='${var.catalog}',
                    schema_name='${var.schema}',
                    table_name_prefix='amount_sum_offline',
                ),
                cron_schedule_trigger=CronSchedule(mode=CronScheduleMode.DERIVED),
            ),
            # Batch online materialized feature.
            MaterializedFeature(
                feature_name='${resources.features.amount_sum.full_name}',
                online_store_config=OnlineStoreConfig(
                    online_store_name='${var.online_store}',
                    catalog_name='${var.catalog}',
                    schema_name='${var.schema}',
                    table_name_prefix='amount_sum_online',
                ),
                cron_schedule_trigger=CronSchedule(mode=CronScheduleMode.DERIVED),
            ),
            # Streaming online materialized feature.
            MaterializedFeature(
                feature_name='${resources.features.amount_sum.full_name}',
                online_store_config=OnlineStoreConfig(
                    online_store_name='${var.online_store}',
                    catalog_name='${var.catalog}',
                    schema_name='${var.schema}',
                    table_name_prefix='amount_sum_online',
                ),
                streaming_mode=StreamingMode(
                    mode=StreamingModeStreamingModeType.STREAMING_MODE_TYPE_RTM
                ),
            ),
        ]
    ),
)

第六步:验证并规划

检查配置,该配置会读取 databricks.yml 以及捆绑包加载的每个资源文件:

databricks bundle validate

validate 也是了解某个字段是否被接受的最便宜方式。 它会命名捆绑不识别的任何字段,包括其路径以及来源的文件和行:

Warning: unknown field: bogus_field
  at resources.features.transaction_count.function.aggregation_function.time_window.sliding
  in resources/batch_feature.yml:16:15

未识别字段会被报告为警告,其值会被丢弃,因此应读取输出而非仅输出代码。 传入 --strict 以将警告视为错误。 关于每个窗口、函数和源接受的字段,请参见 feature。

服务器强制执行的规则,例如带有锯齿窗口的特征视图必须声明所有三个具象特征的规则,在此不检查,且会在databricks bundle deploy上显示。

然后在部署运行前回顾一下具体内容:

databricks bundle plan

在首次部署中,所有七个资源(四个特征视图和三个实体化集)报告 create:

create features.amount_sum
create features.event_count_90m
create features.transaction_count
create features.transaction_count_30d
create materialization_sets.amounts
create materialization_sets.events_online
create materialization_sets.transactions

Plan: 7 to add, 0 to change, 0 to delete, 0 unchanged

每次后续更改都要运行 plan ,看看计划中的行动。

步骤7:部署

将捆绑包部署到工作区:

databricks bundle deploy

部署会在创建每个资源时报告该资源:

Uploading bundle files to /Workspace/Users/someone@example.com/.bundle/feature-store-quickstart/default/files...
Created features.transaction_count_30d
Created features.transaction_count
Created features.amount_sum
Created features.event_count_90m
Created materialization_sets.transactions
Created materialization_sets.events_online
Created materialization_sets.amounts
Files: 4 uploaded, 0 deleted
Resources: 7 created, 0 changed, 0 deleted, 0 unchanged

第八步:检查已创建的内容

databricks bundle summary -o json 报告你的捆绑配置以及上次部署时记录的资源标识符,比如实体化集 ID。 对于 步骤 5 中定义的具体化集:

{
  "id": "1234567890abcdef1234567890abcdef",
  "materialized_features": [
    {
      "cron_schedule_trigger": {
        "mode": "DERIVED"
      },
      "feature_name": "${resources.features.amount_sum.full_name}",
      "offline_store_config": {
        "catalog_name": "main",
        "schema_name": "features_demo",
        "table_name_prefix": "amount_sum_offline"
      }
    },
    {
      "cron_schedule_trigger": {
        "mode": "DERIVED"
      },
      "feature_name": "${resources.features.amount_sum.full_name}",
      "online_store_config": {
        "catalog_name": "main",
        "online_store_name": "my_online_store",
        "schema_name": "features_demo",
        "table_name_prefix": "amount_sum_online"
      }
    },
    {
      "feature_name": "${resources.features.amount_sum.full_name}",
      "online_store_config": {
        "catalog_name": "main",
        "online_store_name": "my_online_store",
        "schema_name": "features_demo",
        "table_name_prefix": "amount_sum_online"
      },
      "streaming_mode": {
        "mode": "STREAMING_MODE_TYPE_RTM"
      }
    }
  ]
}

要查看服务器上存储了什么,可以使用 Feature Engineering 命令将该集合读回,并使用 databricks bundle summary 输出中的 id:

databricks feature-engineering get-materialization-set <materialization-set-id>

响应包含服务器为同一集合存储的内容,包括它生成的字段:

{
  "materialization_set_id": "1234567890abcdef1234567890abcdef",
  "materialized_features": [
    {
      "cron_schedule_trigger": {
        "cron_expression": "0 0 2 * * ?",
        "mode": "DERIVED",
        "timezone_id": "UTC"
      },
      "feature_name": "main.features_demo.amount_sum_3d",
      "is_online": true,
      "job_id": 123456789012345,
      "materialization_set_id": "1234567890abcdef1234567890abcdef",
      "materialized_feature_id": "22222222222222222222222222222222",
      "online_store_config": {
        "catalog_name": "",
        "online_store_name": "my_online_store",
        "schema_name": "",
        "table_name_prefix": ""
      },
      "pipeline_schedule_state": "ACTIVE",
      "table_name": "main.features_demo.amount_sum_online_ceq88t"
    },
    {
      "feature_name": "main.features_demo.amount_sum_3d",
      "is_online": true,
      "job_id": 234567890123456,
      "materialization_set_id": "1234567890abcdef1234567890abcdef",
      "materialized_feature_id": "33333333333333333333333333333333",
      "online_store_config": {
        "catalog_name": "",
        "online_store_name": "my_online_store",
        "schema_name": "",
        "table_name_prefix": ""
      },
      "pipeline_id": "12345678-1234-1234-1234-123456789012",
      "pipeline_schedule_state": "ACTIVE",
      "streaming_mode": {
        "mode": "STREAMING_MODE_TYPE_RTM"
      },
      "table_name": "main.features_demo.amount_sum_online_qiohun"
    },
    {
      "cron_schedule_trigger": {
        "cron_expression": "0 0 2 * * ?",
        "mode": "DERIVED",
        "timezone_id": "UTC"
      },
      "feature_name": "main.features_demo.amount_sum_3d",
      "is_online": false,
      "job_id": 123456789012345,
      "materialization_set_id": "1234567890abcdef1234567890abcdef",
      "materialized_feature_id": "11111111111111111111111111111111",
      "pipeline_schedule_state": "ACTIVE",
      "table_name": "main.features_demo.amount_sum_offline_ceq88t"
    }
  ]
}

步骤9:更改配置

以下示例更改了早期步骤中功能视图和实体化集,并展示了每个变更在部署时的作用。 除非示例中标明了不同的起点,否则每个示例都从 第7步的部署开始。 每个文件都是对声明资源的文件编辑,或者如果在一个文件里声明所有内容,则是对 resources 映射 databricks.yml 的编辑。 在 Python 标签页中,编辑内容会进入 resources/__init__.py。 关于哪些变更是原地应用的,哪些是替换资源的,哪些是被拒绝的,请参见 “改变特征视图 ”和 “改变实体化集”。

移除特征视图中所有具象化的特征

从物化集中移除某个特征视图的所有物化特征这一操作会直接就地执行,因此不会重新计算特征数据。 参见 “更改实体化集”。

以下方法是取transactions第3步定义的特征视图transaction_count和transaction_count_30d,移除特征视图transaction_count_30d中所有具体化特征。 同一集合中 的 transaction_count 具体化特征未被修改:

YAML

 materialization_sets:
   transactions:
     materialized_features:
       # Batch offline materialized feature of transaction_count.
       - feature_name: ${resources.features.transaction_count.full_name}
         offline_store_config: # As in Step 3.
         cron_schedule_trigger:
           mode: DERIVED
       # Batch online materialized feature of transaction_count.
       - feature_name: ${resources.features.transaction_count.full_name}
         online_store_config: # As in Step 3.
         cron_schedule_trigger:
           mode: DERIVED
-      # Batch offline materialized feature of transaction_count_30d.
-      - feature_name: ${resources.features.transaction_count_30d.full_name}
-        offline_store_config: # As in Step 3.
-        cron_schedule_trigger:
-          mode: DERIVED
-      # Batch online materialized feature of transaction_count_30d.
-      - feature_name: ${resources.features.transaction_count_30d.full_name}
-        online_store_config: # As in Step 3.
-        cron_schedule_trigger:
-          mode: DERIVED

Python

 resources.add_materialization_set(
     'transactions',
     MaterializationSet(
         materialized_features=[
             # Batch offline materialized feature of transaction_count.
             MaterializedFeature(...),  # As in Step 3.
             # Batch online materialized feature of transaction_count.
             MaterializedFeature(...),  # As in Step 3.
-            # Batch offline materialized feature of transaction_count_30d.
-            MaterializedFeature(...),  # As in Step 3.
-            # Batch online materialized feature of transaction_count_30d.
-            MaterializedFeature(...),  # As in Step 3.
         ]
     ),
 )

databricks bundle plan 将该集合报告为更新:

update materialization_sets.transactions

Plan: 0 to add, 1 to change, 0 to delete, 6 unchanged

更改特征视图的不可变字段

当实体化集依赖于特征视图时,在部署时更改特征视图中的不可变字段会失败。 替换 Feature View 会先将其删除,且当 Feature View 具有物化特征时,删除会被拒绝。 关于哪些字段是不可变的,以及推荐的更改方式,请参见 “更改特征视图”。

以下内容采用在transaction_count中定义的特征视图,并扩展其滑动窗口时长,而这是一个不可变字段:

YAML

   features:
     transaction_count:
       # The other keys, as in Step 3.
       function:
         aggregation_function:
           count_function:
             input: transaction_id
           time_window:
             sliding:
-              window_duration: 604800s
+              window_duration: 1209600s
               slide_duration: 86400s

Python

 resources.add_feature(
     'transaction_count',
     Feature(
         # The other arguments, as in Step 3.
         function=Function(
             aggregation_function=AggregationFunction(
                 count_function=CountFunction(input='transaction_id'),
                 time_window=TimeWindow(
-                    sliding=SlidingWindow(window_duration='604800s', slide_duration='86400s')
+                    sliding=SlidingWindow(window_duration='1209600s', slide_duration='86400s')
                 ),
             )
         ),
     ),
 )

databricks bundle plan 报告一次重新创建操作:

recreate features.transaction_count

Plan: 1 to add, 0 to change, 1 to delete, 6 unchanged

databricks bundle deploy 请求确认,并说明为何变更无法按当前写法进行:

This action will result in the deletion or recreation of the following Feature Views.
A Feature View that has materialized features in a materialization set cannot be deleted or
recreated. Remove its materialized features from the set in a separate deploy first, then delete
or recreate the Feature View. Deleting or recreating a Feature View that a FeatureSpec serves is
not blocked, and leaves the FeatureSpec stale:
  recreate resources.features.transaction_count

如果您批准,部署将失败,因为集合 transactions 仍保留着 Feature View 的两个物化特征:

Error: cannot recreate resources.features.transaction_count: deleting old id=main.features_demo.transaction_count_7d: Cannot delete feature 'main.features_demo.transaction_count_7d' because it has 2 materialized feature(s). Delete all materialized features first before deleting the feature. (400 BAD_REQUEST)

要实现更改,请部署两次,详见 “更改功能视图”:

  1. 从该集合中移除 Feature View 的实体化特征,并部署。 该集合会在原地更新,比如 移除所有特征视图中具象化的特征。

  2. 更改 Feature View,将其物化特征重新添加到集合中,然后部署。 该计划重建了更改窗口的 Feature View,并重新创建集合,因为向集合添加物化特征会取代它:

    Warn: planning resources.materialization_sets.transactions: the member for feature "main.features_demo.transaction_count_7d" is not in the deployed set, and members cannot be added to an existing set, so the whole set will be recreated: every member is dropped and recomputed
    Warn: planning resources.materialization_sets.transactions: the member for feature "main.features_demo.transaction_count_7d" is not in the deployed set, and members cannot be added to an existing set, so the whole set will be recreated: every member is dropped and recomputed
    recreate features.transaction_count
    recreate materialization_sets.transactions
    
    Plan: 2 to add, 0 to change, 2 to delete, 5 unchanged
    

    批准后,部署会重新创建特征视图,然后创建该集:

    Recreated features.transaction_count
    Recreated materialization_sets.transactions
    Files: 3 uploaded, 0 deleted
    Resources: 2 created, 0 changed, 2 deleted, 5 unchanged
    

更改具体化集的不可变字段

更改集合中任何具象化特征的字段,会替换整个物质化集合,包括编辑未触及的物质化特征。 参见 “更改实体化集”。

以下内容以在transactions中定义的集合为基础,更改其在线目标的table_name_prefix,而该目标是不可变的。 两个在线具体化特征都带有该前缀,因为该集只有一个在线目标:

YAML

 materialization_sets:
   transactions:
     materialized_features:
       # Batch offline materialized feature of transaction_count.
       - feature_name: ${resources.features.transaction_count.full_name}
         offline_store_config: # As in Step 3.
         cron_schedule_trigger:
           mode: DERIVED
       # Batch online materialized feature of transaction_count.
       - feature_name: ${resources.features.transaction_count.full_name}
         online_store_config:
           online_store_name: ${var.online_store}
           catalog_name: ${var.catalog}
           schema_name: ${var.schema}
-          table_name_prefix: transactions_online
+          table_name_prefix: transactions_online_v2
         cron_schedule_trigger:
           mode: DERIVED
       # Batch offline materialized feature of transaction_count_30d.
       - feature_name: ${resources.features.transaction_count_30d.full_name}
         offline_store_config: # As in Step 3.
         cron_schedule_trigger:
           mode: DERIVED
       # Batch online materialized feature of transaction_count_30d.
       - feature_name: ${resources.features.transaction_count_30d.full_name}
         online_store_config:
           online_store_name: ${var.online_store}
           catalog_name: ${var.catalog}
           schema_name: ${var.schema}
-          table_name_prefix: transactions_online
+          table_name_prefix: transactions_online_v2
         cron_schedule_trigger:
           mode: DERIVED

Python

 resources.add_materialization_set(
     'transactions',
     MaterializationSet(
         materialized_features=[
             # Batch offline materialized feature of transaction_count.
             MaterializedFeature(...),  # As in Step 3.
             # Batch online materialized feature of transaction_count.
             MaterializedFeature(
                 feature_name='${resources.features.transaction_count.full_name}',
                 online_store_config=OnlineStoreConfig(
                     online_store_name='${var.online_store}',
                     catalog_name='${var.catalog}',
                     schema_name='${var.schema}',
-                    table_name_prefix='transactions_online',
+                    table_name_prefix='transactions_online_v2',
                 ),
                 cron_schedule_trigger=CronSchedule(mode=CronScheduleMode.DERIVED),
             ),
             # Batch offline materialized feature of transaction_count_30d.
             MaterializedFeature(...),  # As in Step 3.
             # Batch online materialized feature of transaction_count_30d.
             MaterializedFeature(
                 feature_name='${resources.features.transaction_count_30d.full_name}',
                 online_store_config=OnlineStoreConfig(
                     online_store_name='${var.online_store}',
                     catalog_name='${var.catalog}',
                     schema_name='${var.schema}',
-                    table_name_prefix='transactions_online',
+                    table_name_prefix='transactions_online_v2',
                 ),
                 cron_schedule_trigger=CronSchedule(mode=CronScheduleMode.DERIVED),
             ),
         ]
     ),
 )

databricks bundle plan 报告该集合的重建,因此其两个批次离线实体化特征也被丢弃并重新计算:

recreate materialization_sets.transactions

Plan: 1 to add, 0 to change, 1 to delete, 6 unchanged

向现有集合添加物化特征

向已部署的物化集中添加物化特征会替换该集合,因此其中的每个物化特征都会重新计算。 参见 “更改实体化集”。

以下内容获取按照删除特征视图的所有具体化特征所述删除了特征视图 transaction_count_30d 的所有具体化特征之后的 transactions 集,并将特征视图 transaction_count_30d 的实体化特征重新添加回来:

YAML

 materialization_sets:
   transactions:
     materialized_features:
       # Batch offline materialized feature of transaction_count.
       - feature_name: ${resources.features.transaction_count.full_name}
         offline_store_config: # As in Step 3.
         cron_schedule_trigger:
           mode: DERIVED
       # Batch online materialized feature of transaction_count.
       - feature_name: ${resources.features.transaction_count.full_name}
         online_store_config: # As in Step 3.
         cron_schedule_trigger:
           mode: DERIVED
+      # Batch offline materialized feature of transaction_count_30d.
+      - feature_name: ${resources.features.transaction_count_30d.full_name}
+        offline_store_config: # As in Step 3.
+        cron_schedule_trigger:
+          mode: DERIVED
+      # Batch online materialized feature of transaction_count_30d.
+      - feature_name: ${resources.features.transaction_count_30d.full_name}
+        online_store_config: # As in Step 3.
+        cron_schedule_trigger:
+          mode: DERIVED

Python

 resources.add_materialization_set(
     'transactions',
     MaterializationSet(
         materialized_features=[
             # Batch offline materialized feature of transaction_count.
             MaterializedFeature(...),  # As in Step 3.
             # Batch online materialized feature of transaction_count.
             MaterializedFeature(...),  # As in Step 3.
+            # Batch offline materialized feature of transaction_count_30d.
+            MaterializedFeature(...),  # As in Step 3.
+            # Batch online materialized feature of transaction_count_30d.
+            MaterializedFeature(...),  # As in Step 3.
         ]
     ),
 )

databricks bundle plan 会针对每个添加的具体化特征发出警告,并报告需要重新创建:

Warn: planning resources.materialization_sets.transactions: the member for feature "main.features_demo.transaction_count_30d" is not in the deployed set, and members cannot be added to an existing set, so the whole set will be recreated: every member is dropped and recomputed
Warn: planning resources.materialization_sets.transactions: the member for feature "main.features_demo.transaction_count_30d" is not in the deployed set, and members cannot be added to an existing set, so the whole set will be recreated: every member is dropped and recomputed
recreate materialization_sets.transactions

Plan: 1 to add, 0 to change, 1 to delete, 6 unchanged

databricks bundle deploy 在重新创建集合前请求确认:

This action will result in the deletion or recreation of the following materialization sets.
Every materialized feature in a recreated set is dropped and recomputed from scratch, including
the members whose configuration did not change:
  recreate resources.materialization_sets.transactions

移除特征视图中的部分物化特征

从实体化集中移除某个 Feature View 的部分而非全部实体化特征会被拒绝。 参见 “更改实体化集”。

以下内容采用在 步骤5 中为特征视图 amount_sum 定义的集合 amounts,该特征视图具有锯齿窗口,并且仅移除其批量离线物化特征:

YAML

 materialization_sets:
   amounts:
     materialized_features:
-      # Batch offline materialized feature.
-      - feature_name: ${resources.features.amount_sum.full_name}
-        offline_store_config: # As in Step 5.
-        cron_schedule_trigger:
-          mode: DERIVED
       # Batch online materialized feature.
       - feature_name: ${resources.features.amount_sum.full_name}
         online_store_config: # As in Step 5.
         cron_schedule_trigger:
           mode: DERIVED
       # Streaming online materialized feature.
       - feature_name: ${resources.features.amount_sum.full_name}
         online_store_config: # As in Step 5.
         streaming_mode:
           mode: STREAMING_MODE_TYPE_RTM

Python

 resources.add_materialization_set(
     'amounts',
     MaterializationSet(
         materialized_features=[
-            # Batch offline materialized feature.
-            MaterializedFeature(...),  # As in Step 5.
             # Batch online materialized feature.
             MaterializedFeature(...),  # As in Step 5.
             # Streaming online materialized feature.
             MaterializedFeature(...),  # As in Step 5.
         ]
     ),
 )

编辑会移除 amount_sum 的批量离线物化特征,而该删除也会删除同一特征视图中的另外两个物化特征,而编辑并未要求删除这些特征。 部署被拒绝:

Error: cannot update resources.materialization_sets.amounts: updating id=1234567890abcdef1234567890abcdef: Deleting materialized features '11111111111111111111111111111111' of feature 'main.features_demo.amount_sum_3d' also deletes materialized features '22222222222222222222222222222222', '33333333333333333333333333333333', which the request does not delete. Delete the unrequested materialized features as well. (400 INVALID_PARAMETER_VALUE)

第十步:清理

实体化流水线会按照配置的排程继续运行,直到集合被删除。 完成后销毁捆绑包:

databricks bundle destroy

这会删除 bundle 包含的每个文件中声明的实体化集和特征视图。 在最后一个共享实体化特征的实体化特征消失后,后台进程会移除表、管道和作业,因此相关成本停止前会有短暂延迟。 只有创建实体化功能的用户才能删除它,所以要用部署捆绑包的同一身份来运行 databricks bundle destroy 。 请参阅 delete_materialized_feature()。

关于具体化成本及归因方法,请参见 Feature Store 成本管理。

Limitations

  • 流不是捆绑包资源。 在捆绑包之外创建它,并用名称引用它。
  • 将物化特征添加到现有的物化集合中会重新创建该集合,因此该集合中的每个物化特征都会被重新计算。
  • 基于 feature_view_source 或 custom_udf 的特征视图,或使用 request_source 函数的特征视图,无法物化。 请参阅 具体化功能视图。

后续步骤