[Feature] Return context while calling the chat/completions

[mike-bordercon]

Is this your first time submitting a feature request?

• I have searched the existing issues, and I could not find an existing issue for this feature
• I am requesting a straightforward extension of existing functionality

Describe the feature

Hello guys!
We are using canopy in a server-mode.

And it would be super helpful if you add a context which is used in completions into the response for chat/completions endpoint.
It is required to show on UI links/original text which Engine uses to write a completion.

There are two flows to support:

1. non-stream
2. stream

For the first one it's pretty easy just to add a new field:

class ChatResponse(BaseModel):
    ...
    response_context: Optional[Context] = None

And for src.canopy.chat_engine.chat_engine.ChatEngine.chat just add
response.response_context = context

For the "stream" flow, it's pretty breaking as you need to add the whole process as a first chunk.
I'm not a Python-guy, but it could be something like this (in src.canopy_server.app.chat):

            def stringify_content(response: StreamingChatResponse):
                initial_message = {
                    "type": "context",
                    "data": response.response_context.dict() if response.response_context else None
                }
                yield f"data: {json.dumps(initial_message)}\n\n"

                # Then, stream each chunk without the context
                for chunk in response.chunks:
                    chunk.id = question_id 
                    chunk_message = {
                        "type": "chunk",
                        "data": chunk.dict()
                    }
                    yield f"data: {json.dumps(chunk_message)}\n\n"

but maybe it requires a new endpoint.

Describe alternatives you've considered

I can see context/query method, but it will be very untrusted in a chat mode, as you can't predict which text you should put in to receive the same results from Pinecone DB.

Also we can't rely on AI to include required info from context based on prompt, as it sometimes do add it and sometimes now at all.

Who will this benefit?

No response

Are you interested in contributing this feature?

No response

Anything else?

No response