"File name too long" on macOS during catalog backup

[dannykansas]

System & Release Information

> uname -a
Darwin redacted 23.6.0 Darwin Kernel Version 23.6.0: Thu Sep 12 23:35:10 PDT 2024; root:xnu-10063.141.1.701.1~1/RELEASE_ARM64_T6030 arm64

> echo $SHELL
/bin/zsh

> cortex --version
0.27.0

Issue Description

Our Cortex instance is old (2022) and the introduction of CloudControl types - a la type=AWS:S3:Bucket vs. the former type=s3 - has required a fair bit of manual deletion of old types upon discovery.

As part of this process, I take a full instance backup using cortex catalog backup. During the most recent backup, I ran into the following error related to filename length:

OSError: [Errno 63] File name too long: '/Users/myusr/.cortex/export/2025-05-22-16-35-21-default/catalog/arn-aws-ecs-us-west-2-000000000000-task-definition-servc-production-run-ecs-scl-0000-redactl-redact-redact-redac-tred-a-ctredactedredactedredacted-redacted1redacted2redacted3-redacted4redacted5-redacted6redacted77-redacted8redacted9redac-ted10rredacted11redacted12red-acted13redacted14123456789glb-1.yaml'

Redactions above retain the overall length of the filename, for what it's worth.

The full stack trace is as follows:

Traceback (most recent call last):
  File "/opt/homebrew/bin/cortex", line 8, in <module>
    sys.exit(cli())
             ^^^^^
  File "/opt/homebrew/Cellar/cortexapps-cli/0.27.0/libexec/lib/python3.11/site-packages/cortexapps_cli/cortex.py", line 4068, in cli
    args.func(args)
  File "/opt/homebrew/Cellar/cortexapps-cli/0.27.0/libexec/lib/python3.11/site-packages/cortexapps_cli/cortex.py", line 827, in export
    f1 = open(catalog_directory + "/" + output_tag + ".yaml", 'w')
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Suggested Resolution

Ideally, some sort of handling for very long file names in the export would be ideal.

If this is a hard OS limitation, it'd be preferable to simply catch and handle this exception vs. aborting the backup entirely. As of the moment, this fails the backup and I'm unable to safely backup my instance prior to major changes.

[jeff-schnitter]

Hi @dannykansas . Thanks for reporting this. It should definitely be fixed. I'd prefer to shorten the filename instead of reporting it as an error.

Do you find it useful to back up AWS entities? I was thinking of removing them, at least from the default option, since they would get recreated by the Cortex AWS integration so backing up doesn't really seem necessary.

[dannykansas]

@jeff-schnitter That's a good suggestion for a workaround, thank you!

I include these less out of a direct need and more out of a desire to not specify every type by hand with the --types flag:

❯ cortex backup export -h
usage: cortex CLI backup export [-h] [-d] [-t] [-z]

options:
  -h, --help         show this help message and exit
  -d , --directory   Directory where export will be created; defaults to ~/.cortex/export/<DATE>
  -t , --types       Entity types to export. In addition to Cortex x-cortex-type values, this can include scorecard and ip-allowlist. Defaults to all types.
  -z , --pageSize    Number of entities to return in each catalog API call, defaults to 50.

If there were a flag that supported type-level exclusion from an otherwise comprehensive backup, that'd work too.
i.e.: --exclude-types='AWS:RDS:DBInstance','AWS:EC2:Instance' or - even better - something like --exclude-integration-types=aws,azure, that'd be quite helpful.

But for now, I'll make a shell alias that includes the types I truly need backed up so I can minimize typing those out each time.

[jeff-schnitter]

Hi @dannykansas . I'm getting ready to roll out a new version of the CLI and wanted to incorporate your feedback.

Here's the help for the export in this version:

$ cortex backup export -h

 Usage: cortex backup export [OPTIONS]

 Export tenant
 Exports the following objects:
 - catalog
 - entity-types
 - ip-allowlist
 - plugins
 - scorecards
 - workflows

 By default, it does not export any entities that would be created by an integration, for example AWS objects.  This is because these
 entities are maintained by the integration and do not need to be backed up.

 However, these entities can be export by referencing them in the catalog-types parameter, for example this command
 would export all AWS S3 buckets::

 cortex backup export --export-types catalog --catalog-types AWS::S3::Bucket

 It does not back up everything in the tenant.  For example these objects are not backed up:
 - api-keys
 - custom-events
 - custom-metadata created by the public API
 - custom-metrics
 - dependencies created by the API
 - deploys
 - docs created by the API
 - entity-relationships created by the API
 - groups added by the API
 - packages
 - secrets

 In general, if there is a bulk export API method for a Cortex object, it will be included in the export.

╭─ Options ────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ --export-types   -e      TEXT  some help test [default: {'ip-allowlist', 'plugins', 'workflows', 'catalog', 'scorecards', 'entity-types'}]                                                   │
│ --catalog-types  -c      TEXT  Comma separated list of catalog types to export, defaults to service,team,domain plus all user-created entity-types [default: all]                            │
│ --directory      -d      TEXT  Location of export directory, defaults to ~/.cortex/export/<date>-tenant [default: /Users/jeffschnitter/.cortex/export/2025-06-11-15-57-20]                   │
│ --help           -h            Show this message and exit.                                                                                                                                   │
╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯

This should remove the need for your shell alias.

Since you rely on the export, I wanted to see if this planned change would work for you.

[dannykansas]

Hey - apologies for the late reply.

This looks excellent, thank you!