Remove the InSAR workflow intermediate files #65
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
added 22 commits
September 19, 2023 20:40
…into rm_scratch
Contributor
|
Thanks @xhuang-jpl for this PR. Would you please remind me how this PR reduces storage requirements when running InSAR? and would you please document your tests? |
Tyler-g-hudson
approved these changes
Oct 30, 2025
Contributor
Tyler-g-hudson
left a comment
Tyler-g-hudson
left a comment
There was a problem hiding this comment.
LGTM! A couple of small nitpicks, nothing that should block merging
Comment on lines
+68
to
+69
| t_elapsed = time.time() - t_all | ||
| info_channel.log(f"successfully ran prepare_insar_hdf5 in {t_elapsed:.3f} seconds") |
Contributor
There was a problem hiding this comment.
What does this do? These lines seem unrelated to the PR
oberonia78
reviewed
Oct 31, 2025
Contributor
oberonia78
left a comment
oberonia78
left a comment
There was a problem hiding this comment.
Thanks @xhuang-jpl , but I found a bug in the code. Would you be able to fix it?
Comment on lines
120
to
124
| for workflow_name in ['rubbersheet_offsets','ionosphere', 'geo2rdr']: | ||
| workflow_scratch_path = pathlib.Path(f"{scratch_path}/{workflow_name}") | ||
| if workflow_scratch_path.exists() and intermediate_files_removal_flag: | ||
| shutil.rmtree(workflow_scratch_path) | ||
| info_channel.log(f"removed the {workflow_scratch_path} folder") |
Contributor
There was a problem hiding this comment.
Suggested change
| for workflow_name in ['rubbersheet_offsets','ionosphere', 'geo2rdr']: | |
| workflow_scratch_path = pathlib.Path(f"{scratch_path}/{workflow_name}") | |
| if workflow_scratch_path.exists() and intermediate_files_removal_flag: | |
| shutil.rmtree(workflow_scratch_path) | |
| info_channel.log(f"removed the {workflow_scratch_path} folder") | |
| for workflow_name in ['rubbersheet_offsets', 'geo2rdr']: | |
| workflow_scratch_path = pathlib.Path(f"{scratch_path}/{workflow_name}") | |
| if workflow_scratch_path.exists() and intermediate_files_removal_flag: | |
| shutil.rmtree(workflow_scratch_path) | |
| info_channel.log(f"removed the {workflow_scratch_path} folder") |
The ionosphere directory need to be kept until geocode when the ionosphere method is main_diff_ms_band, because the ionosphere in scratch directory is geocoded and store in GUNW.
FYI, I've got error
Traceback (most recent call last):
File "/scratch/jungkyoj/tool/isce3_cuda/src/isce/python/packages/nisar/workflows/insar.py", line 188, in <module>
run(insar_runcfg.cfg, out_paths, persist.run_steps)
File "/scratch/jungkyoj/tool/isce3_cuda/src/isce/python/packages/nisar/workflows/insar.py", line 131, in run
geocode_insar.run(cfg, out_paths['RUNW'], out_paths['GUNW'], InputProduct.RUNW)
File "/scratch/jungkyoj/tool/isce3_cuda/install/packages/nisar/workflows/geocode_insar.py", line 67, in run
cpu_run(cfg, input_hdf5, output_hdf5, input_product_type)
File "/scratch/jungkyoj/tool/isce3_cuda/install/packages/nisar/workflows/geocode_insar.py", line 691, in cpu_run
cpu_geocode_rasters(geocode_obj, geo_datasets, desired,
File "/scratch/jungkyoj/tool/isce3_cuda/install/packages/nisar/workflows/geocode_insar.py", line 501, in cpu_geocode_rasters
get_raster_lists(geo_datasets, desired, freq, pol_list, input_hdf5,
File "/scratch/jungkyoj/tool/isce3_cuda/install/packages/nisar/workflows/geocode_insar.py", line 444, in get_raster_lists
raster, path = get_ds_input_output(
^^^^^^^^^^^^^^^^^^^^
File "/scratch/jungkyoj/tool/isce3_cuda/install/packages/nisar/workflows/geocode_insar.py", line 173, in get_ds_input_output
input_raster = isce3.io.Raster(input_raster_str)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: Error in file /scratch/jungkyoj/tool/isce3_cuda/src/isce/cxx/isce3/io/Raster.cpp, line 28, function isce3::io::Raster::Raster(const std::string&, GDALAccess): failed to create GDAL dataset from file 'HDF5:scratch_dir3/ionosphere/main_diff_ms_band/RUNW.h5://science/LSAR/RUNW/swaths/frequencyB/interferogram/HH/ionospherePhaseScreen'
| baseline.run(cfg, out_paths) | ||
|
|
||
| # Remove the 'bandpass','baseline' scratch folders | ||
| for workflow_name in ['bandpass','baseline']: |
Contributor
There was a problem hiding this comment.
Suggested change
| for workflow_name in ['bandpass','baseline']: | |
| for workflow_name in ['bandpass', 'baseline']: |
nit
|
|
||
| # Remove the 'bandpass','baseline' scratch folders | ||
| for workflow_name in ['bandpass','baseline']: | ||
| workflow_scratch_path = pathlib.Path(f"{scratch_path}/{workflow_name}") |
Contributor
There was a problem hiding this comment.
Suggested change
| workflow_scratch_path = pathlib.Path(f"{scratch_path}/{workflow_name}") | |
| workflow_scratch_path = pathlib.Path(f"{scratch_path}/{workflow_name}") |
nitpick
| baseline.run(cfg, out_paths) | ||
|
|
||
| # Remove the 'bandpass','baseline' scratch folders | ||
| for workflow_name in ['bandpass','baseline']: |
Contributor
There was a problem hiding this comment.
Suggested change
| for workflow_name in ['bandpass','baseline']: | |
| for workflow_name in ['bandpass', 'baseline', 'ionosphere']: |
| troposphere.run(cfg, out_paths['GUNW']) | ||
|
|
||
| # Remove the troposhere scratch folder | ||
| tropo_scratch_path = pathlib.Path(f"{scratch_path}/weather_model_files") |
Contributor
There was a problem hiding this comment.
Suggested change
| tropo_scratch_path = pathlib.Path(f"{scratch_path}/weather_model_files") | |
| tropo_scratch_path = pathlib.Path(f"{scratch_path}/weather_model_files") |
nitpick
| geocode_insar.run(cfg, out_paths['ROFF'], out_paths['GOFF'], InputProduct.ROFF) | ||
|
|
||
| # Remove the geocode scratch folder | ||
| geocode_scratch_path = pathlib.Path(f"{scratch_path}/geocode_corrections") |
Contributor
There was a problem hiding this comment.
Suggested change
| geocode_scratch_path = pathlib.Path(f"{scratch_path}/geocode_corrections") | |
| geocode_scratch_path = pathlib.Path(f"{scratch_path}/geocode_corrections") |
nitpick
|
|
||
| # Remove the 'rubbersheet_offsets','ionosphere', 'geo2rdr' scratch folders | ||
| for workflow_name in ['rubbersheet_offsets','ionosphere', 'geo2rdr']: | ||
| workflow_scratch_path = pathlib.Path(f"{scratch_path}/{workflow_name}") |
Contributor
There was a problem hiding this comment.
Suggested change
| workflow_scratch_path = pathlib.Path(f"{scratch_path}/{workflow_name}") | |
| workflow_scratch_path = pathlib.Path(f"{scratch_path}/{workflow_name}") |
nitpick
| unwrap.run(cfg, out_paths['RIFG'], out_paths['RUNW']) | ||
|
|
||
| # Remove the 'fine_resample_slc','crossmul', 'coarse_resample_slc', 'unwrap' scratch folders | ||
| for workflow_name in ['fine_resample_slc','coarse_resample_slc', |
Contributor
There was a problem hiding this comment.
Suggested change
| for workflow_name in ['fine_resample_slc','coarse_resample_slc', | |
| for workflow_name in ['fine_resample_slc', 'coarse_resample_slc', |
nitpick
| resample_slc_v2.run(cfg, 'fine') | ||
|
|
||
| # Remove the coarse resample scratch folder | ||
| coarse_resample_scratch_path = pathlib.Path(f"{scratch_path}/coarse_resample_slc") |
Contributor
There was a problem hiding this comment.
Suggested change
| coarse_resample_scratch_path = pathlib.Path(f"{scratch_path}/coarse_resample_slc") | |
| coarse_resample_scratch_path = pathlib.Path(f"{scratch_path}/coarse_resample_slc") |
nitpick
added 2 commits
November 3, 2025 18:02
Contributor
Author
|
Thank you @Tyler-g-hudson and @oberonia78 , I have addressed your comments. Please take one more look. |
Contributor
|
Still looks good to me, pending testing and final approval by @oberonia78 |
Contributor
|
@oberonia78 please take another look at this. |
Contributor
Author
|
@oberonia78 If you are ok with this change, would you mind approving it? Thanks |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR is to remove the InSAR workflow intermediate files under the scratch folder. The outputs and dependency of each module can be found here.