Futures

Easy Async

Futures are a powerful tool in Dandy that allow you to run code asynchronously which is great for non-blocking calls like network requests.

All you need to do on any class that has a process method is to call the process_to_future method instead.

Without Async Futures

Let's run three LlmBot.process calls at once and see how long it takes.

Traceback (most recent call last):
  File "/app/.heroku/python/lib/python3.11/site-packages/markdown_exec/_internal/formatters/python.py", line 89, in _run_python
    exec_python(code, code_block_id, exec_globals)
  File "/app/.heroku/python/lib/python3.11/site-packages/markdown_exec/_internal/formatters/_exec_python.py", line 26, in exec_python
    exec(compiled, exec_globals)  # noqa: S102
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<code block: session futures; n1>", line 6, in <module>
    pants_intel = Bot().process('What type of pants should I wear in the rain?')
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/dandy/bot/bot.py", line 79, in process
    return self.llm.prompt_to_intel(**kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/dandy/llm/service.py", line 67, in prompt_to_intel
    return self._llm_connector.prompt_to_intel(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/dandy/llm/connector.py", line 136, in prompt_to_intel
    response_intel_object = self._request_to_intel()
                            ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/dandy/llm/connector.py", line 156, in _request_to_intel
    self._http_request_to_response_str()
  File "/workspace/dandy/llm/connector.py", line 66, in _http_request_to_response_str
    self.response_str = http_connector.request_to_response(
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/dandy/http/connector.py", line 37, in request_to_response
    raise HttpConnectorRecoverableError(message)
dandy.http.exceptions.HttpConnectorRecoverableError: HttpConnector request failed with status code 400 and the following message "Bad Request" and body text "{"error":{"message":"/chat/completions: Invalid model name passed in model=stratus.smart. Call `/v1/models` to view available models for your key.","type":"None","param":"None","code":"400","provider_specific_fields":{"error":"/chat/completions: Invalid model name passed in model=stratus.smart. Call `/v1/models` to view available models for your key."}}}" after 4 attempts

With Async Futures

Now let's run with futures, note you have to access the result attribute of the futures to get to the returned value.

Warning

We recommend you postfix your futures with _future to avoid naming conflicts and confusion.

Traceback (most recent call last):
  File "/app/.heroku/python/lib/python3.11/site-packages/markdown_exec/_internal/formatters/python.py", line 89, in _run_python
    exec_python(code, code_block_id, exec_globals)
  File "/app/.heroku/python/lib/python3.11/site-packages/markdown_exec/_internal/formatters/_exec_python.py", line 26, in exec_python
    exec(compiled, exec_globals)  # noqa: S102
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<code block: session futures; n2>", line 10, in <module>
    print('Pants: ' + pants_intel_future.result.text)
                       ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/dandy/core/future/future.py", line 64, in result
    return self.get_result(self._result_timeout)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/dandy/core/future/future.py", line 52, in get_result
    self._result: R = self._future.result(timeout=self._result_timeout)
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/.heroku/python/lib/python3.11/concurrent/futures/_base.py", line 456, in result
    return self.__get_result()
           ^^^^^^^^^^^^^^^^^^^
  File "/app/.heroku/python/lib/python3.11/concurrent/futures/_base.py", line 401, in __get_result
    raise self._exception
  File "/app/.heroku/python/lib/python3.11/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/dandy/bot/bot.py", line 79, in process
    return self.llm.prompt_to_intel(**kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/dandy/llm/service.py", line 67, in prompt_to_intel
    return self._llm_connector.prompt_to_intel(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/dandy/llm/connector.py", line 136, in prompt_to_intel
    response_intel_object = self._request_to_intel()
                            ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/dandy/llm/connector.py", line 156, in _request_to_intel
    self._http_request_to_response_str()
  File "/workspace/dandy/llm/connector.py", line 66, in _http_request_to_response_str
    self.response_str = http_connector.request_to_response(
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/dandy/http/connector.py", line 37, in request_to_response
    raise HttpConnectorRecoverableError(message)
dandy.http.exceptions.HttpConnectorRecoverableError: HttpConnector request failed with status code 400 and the following message "Bad Request" and body text "{"error":{"message":"/chat/completions: Invalid model name passed in model=stratus.smart. Call `/v1/models` to view available models for your key.","type":"None","param":"None","code":"400","provider_specific_fields":{"error":"/chat/completions: Invalid model name passed in model=stratus.smart. Call `/v1/models` to view available models for your key."}}}" after 4 attempts

Advanced Async

Sometimes in more complex situations you might need to cancel a future or set a timeout.

Traceback (most recent call last):
  File "/app/.heroku/python/lib/python3.11/site-packages/markdown_exec/_internal/formatters/python.py", line 89, in _run_python
    exec_python(code, code_block_id, exec_globals)
  File "/app/.heroku/python/lib/python3.11/site-packages/markdown_exec/_internal/formatters/_exec_python.py", line 26, in exec_python
    exec(compiled, exec_globals)  # noqa: S102
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<code block: session futures; n3>", line 14, in <module>
    print(scary_animal_future.result.text)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/dandy/core/future/future.py", line 64, in result
    return self.get_result(self._result_timeout)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/dandy/core/future/future.py", line 52, in get_result
    self._result: R = self._future.result(timeout=self._result_timeout)
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/.heroku/python/lib/python3.11/concurrent/futures/_base.py", line 456, in result
    return self.__get_result()
           ^^^^^^^^^^^^^^^^^^^
  File "/app/.heroku/python/lib/python3.11/concurrent/futures/_base.py", line 401, in __get_result
    raise self._exception
  File "/app/.heroku/python/lib/python3.11/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/dandy/bot/bot.py", line 79, in process
    return self.llm.prompt_to_intel(**kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/dandy/llm/service.py", line 67, in prompt_to_intel
    return self._llm_connector.prompt_to_intel(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/dandy/llm/connector.py", line 136, in prompt_to_intel
    response_intel_object = self._request_to_intel()
                            ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/dandy/llm/connector.py", line 156, in _request_to_intel
    self._http_request_to_response_str()
  File "/workspace/dandy/llm/connector.py", line 66, in _http_request_to_response_str
    self.response_str = http_connector.request_to_response(
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/dandy/http/connector.py", line 37, in request_to_response
    raise HttpConnectorRecoverableError(message)
dandy.http.exceptions.HttpConnectorRecoverableError: HttpConnector request failed with status code 400 and the following message "Bad Request" and body text "{"error":{"message":"/chat/completions: Invalid model name passed in model=stratus.smart. Call `/v1/models` to view available models for your key.","type":"None","param":"None","code":"400","provider_specific_fields":{"error":"/chat/completions: Invalid model name passed in model=stratus.smart. Call `/v1/models` to view available models for your key."}}}" after 4 attempts