chore: replace legacy datetime rebase tests with current scan coverage [iceberg] (#3605)

Author: andygrove

docs/source/contributor-guide/parquet_scans.md

@@ -49,10 +49,10 @@ The following features are not supported by either scan implementation, and Come
 
 The following shared limitation may produce incorrect results without falling back to Spark:
 
-- No support for datetime rebasing detection or the `spark.comet.exceptionOnDatetimeRebase` configuration. When
-  reading Parquet files containing dates or timestamps written before Spark 3.0 (which used a hybrid
-  Julian/Gregorian calendar), dates/timestamps will be read as if they were written using the Proleptic Gregorian
-  calendar. This may produce incorrect results for dates before October 15, 1582.
+- No support for datetime rebasing. When reading Parquet files containing dates or timestamps written before
+  Spark 3.0 (which used a hybrid Julian/Gregorian calendar), dates/timestamps will be read as if they were
+  written using the Proleptic Gregorian calendar. This may produce incorrect results for dates before
+  October 15, 1582.
 
 The `native_datafusion` scan has some additional limitations, mostly related to Parquet metadata. All of these
 cause Comet to fall back to Spark.

spark/src/test/scala/org/apache/comet/parquet/ParquetReadSuite.scala

@@ -1815,6 +1815,35 @@ class ParquetReadV1Suite extends ParquetReadSuite with AdaptiveSparkPlanHelper {
     })
   }
 
+  test("reading ancient dates before 1582") {
+    // Verify that legacy dates (before 1582-10-15) are read without error.
+    // Comet does not support datetime rebasing, so these dates are read as if they were
+    // written using the Proleptic Gregorian calendar (no rebase, no exception).
+    val file =
+      getResourceParquetFilePath("test-data/before_1582_date_v3_2_0.snappy.parquet")
+
+    Seq(CometConf.SCAN_NATIVE_ICEBERG_COMPAT, CometConf.SCAN_NATIVE_DATAFUSION).foreach {
+      scanImpl =>
+        withSQLConf(CometConf.COMET_NATIVE_SCAN_IMPL.key -> scanImpl) {
+          val df = spark.read.parquet(file)
+
+          // Verify Comet scan is in the plan
+          val plan = df.queryExecution.executedPlan
+          checkCometOperators(plan)
+
+          // Verify all 8 rows are read and contain dates before 1582
+          val rows = df.collect()
+          assert(rows.length == 8, s"Expected 8 rows with $scanImpl, got ${rows.length}")
+          rows.foreach { row =>
+            val date = row.getDate(0)
+            assert(
+              date.toLocalDate.getYear < 1582,
+              s"Expected date before 1582 with $scanImpl, got $date")
+          }
+        }
+    }
+  }
+
 }
 
 // ignored: native_comet scan is no longer supported