Auto Loader & Schema Drift

For those using Databricks Auto Loader ( cloudFiles ), how do you handle schema inference and evolution without breaking downstream ML models? If a new feature column drops in or an upstream data type silently widens, do you rely on the _rescued_data column to catch anomalies, or does the automatic stream restart cause unexpected issues for your online serving pipelines? How does BigQuery handle this kind of raw file ingestion drift by comparison? submitted by /u/Professional-Bowl890 [link] [comments]