problems with Statement Selection

[nickums]

I have trained my chatbot solely with my own input, and specifying read_only = True to prevent new inputs being added.
I am using my own 'get_best_response' algorithm. I am experiencing problems with input selection:-
My own 'get_best_input_match' is not being called
New input statements are sometimes included in the input matching and matched to themselves which results in a default/random response, which draws on a set of statements that I have NOT provided in training.
Here is my code and output:
jokes = [{"id":299,"type":"general","setup":"Where do hamburgers dance?",
"punchline":"The meat-ball."},
{"id":330,"type":"general","setup":"Why did the belt go to prison?","punchline":
"He held up a pair of pants!"}]
for joke in jokes: print(f"setup:{joke['setup']} \n punchline:{joke['punchline']}")

from random import choice
import chatterbot
from chatterbot import ChatBot
from chatterbot.comparisons import levenshtein_distance
from chatterbot.comparisons import synset_distance
from chatterbot.comparisons import jaccard_similarity
import logging
logging.basicConfig(level=logging.INFO) # Enable info level logging
logger = logging.getLogger(name)

def get_best_input_match(s1,s2): # find which algo gives reasonable result
S = synset_distance(s1,s2)
L = synset_distance(s1,s2)
J = synset_distance(s1,s2)
logger.info(f'Syn,Lev,Jcc similarities are {S,L.J}')
return max(S,max(L,J))

def get_best_response(input_statement, response_list, storage=None):
i, best = 0, 0
for response in response_list:
p = response.confidence
if p > best: best, k = p, i
i += 1
if best == 0:
logger.info(f'Selecting no confidence random response.')
return choice(response_list) # random response from the selection
else:
logger.info(f'Selecting response {k} with highest confidence {best}')
return response_list[best]

def chat(input):
response = chatter.get_response(input)
print(f'{input}\n{response.confidence}:{response.text}\n')

chatter = ChatBot("The Joker", read_only = True, # dont learn/add new stmts
response_selection_method = get_best_response,
statement_comparision_function = get_best_input_match)

from chatterbot.trainers import ListTrainer
trainer = ListTrainer(chatter)

ONLY train the jokes - but there are a lot of statements already in the dB?

for item in jokes:
joke = [item['setup'], item['punchline']]; print(joke)
trainer.train(joke)

chat('Why did the belt go to prison?') # T 1 -> matches out of 11(!) optimal responses
chat('why is the belt in prison?') # S1 .82 -> bestmatch Train stmt T -> OK
chat('why was the belt imprisoned?') # S2 .72 matches itself and T -> OK
chat('the belt was imprisoned for what offence?') # S3 searches T S1 S3 matches S3 -> random
chat('why prosecute the belt?') # S4 .53 matches T to S4 -> OK

output:
!
['Where do hamburgers dance?', 'The meat-ball.']

List Trainer: [########## ] 50%
List Trainer: [####################] 100%

['Why did the belt go to prison?', 'He held up a pair of pants!']

List Trainer: [########## ] 50%
List Trainer: [####################] 100%

INFO:chatterbot.chatterbot:Beginning search for close text match
INFO:chatterbot.chatterbot:Processing search results
INFO:chatterbot.chatterbot:Similar text found: Someone said they're going to stop allowing machines in speedrun competitions. Search me why, they'd have to drop every tool-assisted speedrun out there. 0.22
INFO:chatterbot.chatterbot:Similar text found: i always say, if you see an ass go by, kiss it. 0.36
INFO:chatterbot.chatterbot:Similar text found: Why did the belt go to prison? 1.0
INFO:chatterbot.chatterbot:Using "Why did the belt go to prison?" as a close match to "Why did the belt go to prison?" with a confidence of 1.0
INFO:chatterbot.chatterbot:Selecting response from 11 optimal responses.
INFO:main:Selecting no confidence random response.
INFO:chatterbot.chatterbot:Response selected. Using "He held up a pair of pants!"
INFO:chatterbot.chatterbot:BestMatch selected "He held up a pair of pants!" as a response with a confidence of 1.0
Why did the belt go to prison?
1.0:He held up a pair of pants!

INFO:chatterbot.chatterbot:Beginning search for close text match
INFO:chatterbot.chatterbot:Processing search results
INFO:chatterbot.chatterbot:Similar text found: Why did the belt go to prison? 0.82
INFO:chatterbot.chatterbot:Similar text found: why was the belt in prison? 0.94
INFO:chatterbot.chatterbot:Using "Why did the belt go to prison?" as a close match to "why is the belt in prison?" with a confidence of 0.82
INFO:chatterbot.chatterbot:Selecting response from 11 optimal responses.
INFO:main:Selecting no confidence random response.
INFO:chatterbot.chatterbot:Response selected. Using "He held up a pair of pants!"
INFO:chatterbot.chatterbot:BestMatch selected "He held up a pair of pants!" as a response with a confidence of 0.82
why is the belt in prison?
0.82:He held up a pair of pants!

INFO:chatterbot.chatterbot:Beginning search for close text match
INFO:chatterbot.chatterbot:Processing search results
INFO:chatterbot.chatterbot:Similar text found: Why did the belt go to prison? 0.72
INFO:chatterbot.chatterbot:Similar text found: why was the belt in prison? 0.91
INFO:chatterbot.chatterbot:Using "Why did the belt go to prison?" as a close match to "why was the belt imprisoned?" with a confidence of 0.72
INFO:chatterbot.chatterbot:Selecting response from 11 optimal responses.
INFO:main:Selecting no confidence random response.
INFO:chatterbot.chatterbot:Response selected. Using "He held up a pair of pants!"
INFO:chatterbot.chatterbot:BestMatch selected "He held up a pair of pants!" as a response with a confidence of 0.72
why was the belt imprisoned?
0.72:He held up a pair of pants!

INFO:chatterbot.chatterbot:Beginning search for close text match
INFO:chatterbot.chatterbot:Processing search results
INFO:chatterbot.chatterbot:Similar text found: Why did the belt go to prison? 0.48
INFO:chatterbot.chatterbot:Similar text found: why was the belt in prison? 0.5
INFO:chatterbot.chatterbot:Similar text found: the belt imprisoned for what offence? 0.95
INFO:chatterbot.chatterbot:Using "the belt imprisoned for what offence?" as a close match to "the belt was imprisoned for what offence?" with a confidence of 0.95
INFO:chatterbot.chatterbot:No responses found. Generating alternate response list.
INFO:chatterbot.chatterbot:No known response to the input was found. Selecting a random response.
INFO:chatterbot.chatterbot:BestMatch selected "Someone said they're going to stop allowing machines in speedrun competitions. Search me why, they'd have to drop every tool-assisted speedrun out there." as a response with a confidence of 0
the belt was imprisoned for what offence?
0:Someone said they're going to stop allowing machines in speedrun competitions. Search me why, they'd have to drop every tool-assisted speedrun out there.

INFO:chatterbot.chatterbot:Beginning search for close text match
INFO:chatterbot.chatterbot:Processing search results
INFO:chatterbot.chatterbot:Similar text found: Why did the belt go to prison? 0.53
INFO:chatterbot.chatterbot:Similar text found: why was the belt in prison? 0.6
INFO:chatterbot.chatterbot:Using "Why did the belt go to prison?" as a close match to "why prosecute the belt?" with a confidence of 0.53
INFO:chatterbot.chatterbot:Selecting response from 11 optimal responses.
INFO:main:Selecting no confidence random response.
INFO:chatterbot.chatterbot:Response selected. Using "He held up a pair of pants!"
INFO:chatterbot.chatterbot:BestMatch selected "He held up a pair of pants!" as a response with a confidence of 0.53
why prosecute the belt?
0.53:He held up a pair of pants!