Merge pull request #48 from lsst/tickets/SP-2474

tickets/SP-2474: Fix data storage issue in custom coadds notebook

Author: ameisner

DP1/100_How_to_Use_RSP_Tools/105_Image_reprocessing/105_6_Create_custom_coadd_images.ipynb

@@ -19,7 +19,7 @@
     "Data Release: DP1 <br>\n",
     "Container Size: large <br>\n",
     "LSST Science Pipelines version: r29.1.1 <br>\n",
-    "Last verified to run: 2025-06-24 <br>\n",
+    "Last verified to run: 2025-07-29 <br>\n",
     "Repository: <a href=\"https://github.com/lsst/tutorial-notebooks\">github.com/lsst/tutorial-notebooks</a> <br>"
    ]
   },
@@ -82,6 +82,7 @@
    "outputs": [],
    "source": [
     "import os\n",
+    "from pathlib import Path\n",
     "from lsst.daf.butler import Butler\n",
     "from lsst.ctrl.mpexec import SimplePipelineExecutor\n",
     "from lsst.pipe.base import Pipeline\n",
@@ -543,7 +544,7 @@
    "id": "85394b0f-7222-4624-89af-4c1f9c70d7d8",
    "metadata": {},
    "source": [
-    "Users cannot write to the main DP1 Butler repository. Instead, make a writable local repository within your current directory where custom coadd outputs will be written. The call to `use_local_butler` creates a new directory that will contain the local Butler repository, within your current working directory."
+    "Users cannot write to the main DP1 Butler repository. Instead, make a writable repository where custom coadd outputs will be written. The call to `use_local_butler` creates a new directory that will contain the local Butler repository, within the `$SCRATCH_DIR` temporary space that gets deleted weekly. Use this temporary space because the custom coadd processing produces of order a gigabyte of intermediate files."
    ]
   },
   {
@@ -553,8 +554,8 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "local_repo_name = os.path.join(os.getenv(\"HOME\"), \"my_local_repo\")\n",
-    "out_butler = executor.use_local_butler(local_repo_name)"
+    "local_repo_path = Path(os.getenv(\"SCRATCH_DIR\"), \"my_local_repo\")\n",
+    "out_butler = executor.use_local_butler(local_repo_path)"
    ]
   },
   {
@@ -634,13 +635,107 @@
     "> Figure 2: The custom `deep_coadd` image using only the selected subset of input visit images."
    ]
   },
+  {
+   "cell_type": "markdown",
+   "id": "a95de02e-d676-4ca3-8fb8-d1afe410e1d4",
+   "metadata": {},
+   "source": [
+    "### 3.4. Copy the custom coadd output file"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "aa9eacc5-18b0-4a8a-aa05-4b45c12aeee8",
+   "metadata": {},
+   "source": [
+    "The writable Butler repository used for custom coaddition is located within a temporary storage area. The Butler `retrieveArtifacts` method can be used to retrieve a copy of just the custom coadd output file, without copying intermediate files generated by the processing, and placing the custom coadd output file in a desired location. This can be useful to preserve the custom coadd image file without filling up a significant amount of disk quota in a permanent location. Start by retrieving the dataset reference for the custom coadd image output file."
+   ]
+  },
   {
    "cell_type": "code",
    "execution_count": null,
    "id": "91ee3647-96c9-4aa9-9d61-ced39b8c1990",
    "metadata": {},
    "outputs": [],
-   "source": []
+   "source": [
+    "refs = out_butler.query_datasets('deep_coadd_predetection',\n",
+    "                                 collections=executor.quantum_graph.metadata[\"output_run\"])"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "f5dee19c-5bc9-4f5b-af9c-aaabfec6ee77",
+   "metadata": {},
+   "source": [
+    "As an example, choose to put the copy of the custom coadd output file in your scratch space area."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "9e55d973-a30d-4891-a896-6b62e6b08a9c",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "outdir = os.getenv(\"SCRATCH_DIR\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "e639644d-14e2-47ba-92d1-03a952f634c2",
+   "metadata": {},
+   "source": [
+    "*Option*: instead choose to copy the custom coadd output file to your home directory, where it will not be deleted within a week."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "e0f81548-e8c0-4854-b2f7-767da392f27b",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# outdir = os.getenv(\"HOME\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "1fdd59a0-0454-441d-b60f-4d6aeb10d147",
+   "metadata": {},
+   "source": [
+    "Copy the FITS file containing the custom coadd image to the desired location."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "a3913c72-355a-4289-8061-b22785ff7df9",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "out_butler.retrieveArtifacts(refs, outdir, preserve_path=False, transfer=\"copy\", overwrite=True)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "044a5afe-202b-4bbd-9c96-91869f89716d",
+   "metadata": {},
+   "source": [
+    "To free up space, delete the writable Butler repository used for custom coaddition. If you wish to continue exploring the writable Butler repository then the following cell can be skipped, though note that the local Butler repo will still be deleted within a week because it is within temporary scratch space. In general, be very careful about any `rm -r` or similar commands that you run on RSP, as there is potential for inadvertently deleting files you wish to retain.\n",
+    "\n",
+    "*Option*: Skip the final cell below if you wish to continue working with variables previously defined in this notebook and/or continue using your writable Butler repo."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "f9c8a16c-ed9e-46f7-81c2-43ea4206dc0d",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "%reset -f\n",
+    "! rm -r $SCRATCH_DIR/my_local_repo"
+   ]
   }
  ],
  "metadata": {