What notion of similarity is TLSH preserving?

[a-gardner1]

I'm having a tough time evaluating whether TLSH is appropriate for my application. I expect TLSH to correlate to some other semantically meaningful notion of document similarity such as Hamming or Levenshtein distance, but some simple experiments with random strings indicate that is not the case. Namely, I found that for pairs taken from 1000 randomly chosen strings each of 200 characters, the Pearson correlation coefficient between TLSH distances (i.e., tlsh.diff) and either (bitwise) Hamming or (char-wise) Levenshtein distances ranged from 0.14 to 0.18, approximately. Meanwhile, the Hamming and Levenshtein distances has a Pearson correlation of approximately 0.72.

Is there a conventional distance or similarity metric that TLSH correlates with?

Edit: I forgot to remove dependent distance values from swapping the order of arguments when calculating the correlations. Doing so reduced the TLSH correlation to near zero and the Hamming/Levenshtein correlation to approximately 0.18.

Edit 2: It also occurs to me that perhaps the correlation depends upon the magnitude of the distances, i.e., they are correlated for small distances but not for large. The random generation method above was biased towards large distances; I'll try to investigate this thread when I have time.

[jonjoliver]

It is optimizing the distance against typical attacks in cyberattacks.
The ROC curve optimization is described in Fig1 of https://github.com/trendmicro/tlsh/blob/master/TLSH_CTC_final.pdf
The types of modifications are described in https://github.com/trendmicro/tlsh/blob/master/Attacking_LSH_and_Sim_Dig.pdf
They include

• mutation of spam images
• swapping words in text / deleting words / swapping words / etc etc
• modifying source code so that the code has the same functionality (example swapping 2 functions)
• mutating HTML
• mutations in malware
• mutations in word docs / excel docs

Your experiment is quite biased towards very large levels of change.
You want to detect modest - medium change.
Comparing large levels of change is not that meaningful as long as you get a distance score that says samples are very far apart.

Also edit distance absolute will fail at tasks such as swapping the order of functions in code and recompiling.
Example
code1 has functions A B C D E F
code2 has functions F E D C B A

edit distance will say they are far apart - even though from a semantic point of view they are almost identical.
TLSH will do much better at this task than edit distance.

[a-gardner1]

Thanks for the clarification. I haven't had a chance to explore this further, but I'd like to give some extra context for the question.

In short, we are trying to perform metric learning with a neural network so that when one embeds two images, say both of the same digit in the MNIST handwritten character dataset, then the TLSH distance between the embedded images is small. For images of different digits, the TLSH distance between their embedding should be large. We had been getting pretty poor results and were trying to gain some intuition for why we weren't getting the desired distance pattern.