Just-in-time refresh and intelligent polling for external catalog metadata (private preview)¶

On your account, just-in-time (JIT) refresh and intelligent polling work together on supported externally managed Apache Iceberg™ tables to keep catalog metadata fresh with less wasted catalog traffic than fixed-interval background polling alone.

After you explicitly acknowledge this preview and your account team enables it on your account, intelligent polling is on by default. You do not configure anything to turn it on. When intelligent polling is active, Snowflake no longer respects REFRESH_INTERVAL_SECONDS for background polling cadence. Snowflake plans to deprecate that parameter in favor of event-driven notification or intelligent polling.

Just-in-time refresh enhances automatic refresh. Writes to eligible tables always refresh metadata at query time. On reads, ICEBERG_METADATA_CACHE_SECONDS bounds how stale catalog metadata may be before a query triggers a query-time refresh.

Intelligent polling drives how often Snowflake checks the external catalog for metadata updates in the background. Snowflake uses machine learning to predict when each table is likely to receive a new snapshot and polls closer to those windows. Tables are refreshed closer to the frequency of the updates they receive, with the goal of keeping metadata fresh most of the time.

Parameter names, defaults, and supported workloads can change during the preview. Contact your account team to report feedback or escalate issues.

Eligibility¶

JIT refresh and intelligent polling apply to tables that use any of the following:

• Iceberg REST catalog integration — Snowflake connects to a remote catalog through the Apache Iceberg REST OpenAPI specification. Examples include Unity Catalog, Tabular, Apache Polaris™ REST endpoints, and AWS Glue Iceberg REST.
• Open Catalog catalog integration — Snowflake connects to Snowflake Open Catalog, which also uses a REST interface.
• Native AWS Glue catalog integration — Snowflake connects to the AWS Glue Data Catalog directly.

They do not apply to Snowflake-managed Iceberg, object-storage catalog integrations, or other ineligible types unless your account team states otherwise.

Overview¶

Externally managed Iceberg tables rely on auto-refresh: Snowflake periodically polls your external catalog for metadata changes and refreshes when it detects updated table metadata.

On your account, background polling uses intelligent polling instead of a single fixed interval for every table. Snowflake adapts how often it checks the catalog per table according to how often each table tends to change. That avoids the tradeoff of one global REFRESH_INTERVAL_SECONDS value:

• A short fixed interval would improve freshness for hot tables but flood the catalog with requests, including for tables that rarely change.
• A long fixed interval would lower catalog traffic but increase lag between external commits and what Snowflake queries return.

JIT refresh handles read-side freshness: ICEBERG_METADATA_CACHE_SECONDS bounds how stale metadata may be before a SELECT triggers a query-time refresh. Intelligent background polling still runs so metadata is often already fresh at query time, which reduces how often read paths must refresh during the query.

Common reasons to tune this preview include:

• Reducing unpredictable latency after external writers commit new snapshots.
• Setting read freshness with ICEBERG_METADATA_CACHE_SECONDS without choosing one polling interval for every table.
• Monitoring whether intelligent polling keeps catalog traffic efficient for your workload.

How JIT and intelligent polling fit with automatic refresh¶

Snowflake uses both intelligent background polling and read-side JIT. Background intelligent polling tries to reduce lag and keep metadata fresh so queries do not have to pay the cost of a full metadata refresh at execution time. JIT enforces your freshness bound when cached metadata is too old.

ICEBERG_METADATA_CACHE_SECONDS¶

This parameter sets, in seconds, how long Snowflake may treat Iceberg catalog metadata as fresh on the read path before a SELECT (or other supported read) triggers a JIT refresh for an eligible table.

Allowed values: -1, or integers 0 through 86400 (one day), inclusive. -1 is the only negative value allowed.

Default: 300 seconds at table lineage when no value is inherited from a higher scope (JIT refresh is on with that cache duration).

Where to set: CREATE / ALTER ICEBERG TABLE, CREATE / ALTER SCHEMA, CREATE / ALTER DATABASE, or ALTER ACCOUNT.

Precedence (most specific wins): table → schema → database → account.

Defaults set at schema, database, or account level apply only to supported tables (Iceberg REST catalog integration, Open Catalog catalog integration, or native AWS Glue catalog integration) in that scope. Setting the parameter on a non-eligible table fails.

Semantics (reads only)

• -1 — JIT refresh is off for that table on reads. Intelligent background polling still applies.
• 0 — on reads, always refresh metadata at query time if the table supports JIT refresh.
• Greater than 0 — on reads, bound how stale catalog-backed metadata may be before Snowflake refreshes it at query time.

Examples

Table-level:

ALTER ICEBERG TABLE my_ext_table SET ICEBERG_METADATA_CACHE_SECONDS = 5;

Schema default:

ALTER SCHEMA my_schema SET ICEBERG_METADATA_CACHE_SECONDS = 10;

Database-level:

ALTER DATABASE my_linked_db SET ICEBERG_METADATA_CACHE_SECONDS = 0;

Turn JIT refresh off for a table on reads:

ALTER ICEBERG TABLE my_ext_table SET ICEBERG_METADATA_CACHE_SECONDS = -1;

Account default:

ALTER ACCOUNT SET ICEBERG_METADATA_CACHE_SECONDS = 300;

Use the same parameter name on CREATE ICEBERG TABLE, CREATE SCHEMA, CREATE DATABASE, and account-level configuration where supported.

Statement types during the preview¶

ICEBERG_METADATA_CACHE_SECONDS affects reads (primarily SELECT, including joins, CTEs, and unions). During private preview, which read shapes are supported may still vary by account; confirm with your account team before you rely on JIT for a given query pattern.

Intelligent polling¶

What intelligent polling does¶

After preview enablement, intelligent polling is on by default. You do not set a parameter or catalog-integration option to enable it.

Snowflake uses machine learning to predict future snapshot-creation times from past snapshot timestamps, then schedules catalog checks around those predicted windows. The scheduler optimizes to reduce lag (the time between when a snapshot is created in the catalog and when Snowflake detects it) as much as possible.

Prediction and scheduling loop¶

Intelligent polling runs in a feedback loop:

1. From polled tables, Snowflake collects snapshot creation timestamps and uses them in a machine learning algorithm to predict future snapshot creation times.
2. Tables with predicted snapshot times in the upcoming interval are placed on a list of tables to poll soon.
3. Auto-refresh polls the tables on that list.

Behaviors to expect¶

• Cold start — A table new to intelligent polling (just created in the catalog or newly linked in Snowflake) is not polled immediately. Instead, Snowflake schedules it for polling more frequently until there is enough history to establish heuristics. Heuristics and change predictions are not available for at least 30 minutes after the table is created.
• Quiet tables — Every table is polled at least once within a long minimum horizon (the exact interval is not finalized for the preview). This covers rarely changed tables that the model might otherwise deprioritize.
• No customer tuning knob for cadence — You do not steer background polling with REFRESH_INTERVAL_SECONDS. Read-side freshness is controlled with ICEBERG_METADATA_CACHE_SECONDS.

REFRESH_INTERVAL_SECONDS (catalog integration)¶

REFRESH_INTERVAL_SECONDS may still appear on your catalog integration, but Snowflake ignores it for background intelligent polling. Changing it does not alter per-table adaptive cadence.

Snowflake plans to deprecate REFRESH_INTERVAL_SECONDS in favor of event-driven notification or intelligent polling. Until that happens, treat the parameter as unused for background cadence on accounts where intelligent polling is enabled.

Read-side JIT at query time is separate and is off on reads when the effective ICEBERG_METADATA_CACHE_SECONDS for a table is -1.

If your runbooks or monitoring assume that shortening REFRESH_INTERVAL_SECONDS increases background freshness, revisit those assumptions. Snowflake controls background cadence through intelligent polling.

Observability¶

During private preview, use the same surfaces as standard Iceberg auto-refresh:

• SYSTEM$AUTO_REFRESH_STATUS — Check pipe health with fields such as executionState and pendingSnapshotCount.
• Event table logging for Iceberg auto-refresh — Existing auto-refresh events continue to apply.

Intelligent polling does not add new status fields or event types in this preview. Watch pending snapshots and wall-clock lag (time from an external commit until a SELECT returns the new data) when you evaluate freshness.

Example:

SELECT SYSTEM$AUTO_REFRESH_STATUS('my_db.my_schema.my_iceberg_table');

Testing guidance¶

The scenarios below are optional. They are not a required test plan. Use what fits your workload, change one variable at a time when you compare results, and contact your account team when you want help or want to share feedback.

If something looks wrong, a query_id and a short note of table configuration (including ICEBERG_METADATA_CACHE_SECONDS and catalog integration name) are often enough for follow-up.

JIT refresh scenarios¶

Read-side JIT follows ICEBERG_METADATA_CACHE_SECONDS. Background polling follows intelligent polling rules above.

• External write, then read — After a commit outside Snowflake, measure wall-clock lag until a SELECT sees the new data with ICEBERG_METADATA_CACHE_SECONDS set to 0, a small positive value, and the account or table default.
• Multi-table reads — Joins, unions, or CTEs over multiple JIT-enabled tables, especially when only one upstream table changed.
• Cache-duration tuning — Vary only ICEBERG_METADATA_CACHE_SECONDS to isolate read-side JIT from background intelligent polling.

Intelligent polling scenarios¶

Focus on pending snapshots (pendingSnapshotCount from SYSTEM$AUTO_REFRESH_STATUS) and wall-clock lag after external commits:

• External write, then observe — After a commit outside Snowflake, check whether pendingSnapshotCount clears and how long until a SELECT returns the new data.
• Frequently vs infrequently updated tables — Side-by-side tables on the same catalog integration. Hot tables should see tighter wall-clock lag; quiet tables should still refresh eventually without matching the hot table’s cadence.

What to look out for¶

Brief notes on what you tried and what stood out—and query_id values only for cases you want investigated—are usually enough for preview feedback.

Limitations and expectations¶

• After a table is created, intelligent-polling heuristics are not established for at least 30 minutes, and snapshot changes are not predicted for at least 30 minutes.
• Dynamic Tables that read from JIT-enabled Iceberg tables do not respect ICEBERG_METADATA_CACHE_SECONDS, so you may see stale data.
• JIT refresh and intelligent polling support Iceberg REST catalog integrations, Open Catalog catalog integrations, and native AWS Glue catalog integrations. They do not support Snowflake-managed Iceberg, object-storage catalog integrations, or other ineligible types unless your account team states otherwise.
• Preview features are not committed to a specific region, cloud, edition, or GA date.
• Prefer non-production testing unless your organization accepts preview risk.