Wrong tasks routing - DEFAULT_VERSION_HOSTNAME env variable set to empty string

[ntassev]

When a task is added from a Flex VM service using the python-compat runtime, wrong target Host header is being calculated forcing the task to be executed on the same version and the same module, regardless of the target parameter of the taskqueue.add method.

This seems to be caused by app_identity.get_default_version_hostname returning empty string which in turn happens because DEFAULT_VERSION_HOSTNAME environment variable is set to a default empty string and not populated anywhere. I am surprised this is not handled since there is code in the python-compat runtime that takes care to initialize the env in a compatible way with GAE. Am I missing something?

Workaround for me currently is to monkey-patch app_identity.get_default_version_hostname to return the value of GAE_APPENGINE_HOSTNAME environment variable which is correctly set on Flex VM when enable_app_engine_apis: true flag is set in the YAML descriptor of the service.

[duggelz]

Thanks for filing this issue.

Can you tell me whether your app.yaml file has vm: true or env: flex? The first indicates that you are using the Managed VM environment, the second indicates that you're using the App Engine Flexible environment. The names and contents of many environment variables changed between those two environments, and it sounds like the code may not handle both.

Please note that the beta Managed VMs environment is deprecated in favor of the App Engine Flexible environment, which is GA. And this beta python-compat runtime is deprecated in favor of the python runtime, which is also GA. See Managed VMs upgrade guide for more information.

[ntassev]

It is env:flex - here is the excerpt from the descriptor some-service.yaml (it is not the default, hence not app.yaml. The default module and other services are not on VMs but on GAE classic):

service: some-service
runtime: python-compat
env: flex
entrypoint: gunicorn -b :$PORT main:app
runtime_config:
  python_version: 2
beta_settings:
  enable_app_engine_apis: true

It used to be vm:true, but we migrated to take advantage of the extra time while python-compat will be available (until October). We have code that depends on NDB and taskqueues, so the python Flex runtime is not an option right now.

I checked the env variables available both in the container and during processing an HTTP request and DEFAULT_VERSION_HOSTNAME is always an empty string.

[ntassev]

Speaking more broadly on the topic, we're using the following shims, where we discovered some fluctuations in behavior between the four runtimes - GAE classic, Managed VM vm:true, Flex with python-compat and Flex with python I'm not 100% sure they are completely accurate, but I'd be glad if you or somebody comments on them or if they'd help somebody facing migration issues.

def is_gae_classic():
    return ('GCLOUD_PROJECT' not in os.environ) and ('GAE_VM' not in os.environ)


def is_flex_env():
    return 'GCLOUD_PROJECT' in os.environ


def get_current_module_name():
    try:
        # works only on GAE proper
        from google.appengine.api import modules
        name = modules.get_current_module_name()
        if not name:
            raise Exception()
        return name
    except:
        # for `vm:true` and `env:flex python-compat`
        if 'GAE_MODULE_NAME' in os.environ:
            return os.environ.get('GAE_MODULE_NAME')
        else:
            # on new `env:flex`
            return os.environ['GAE_SERVICE']


def get_application_id():
    try:
        # works on: GAE proper, `vm:true`, `env:flex python-compat`
        from google.appengine.api import app_identity
        return app_identity.get_application_id()
    except:
        if 'GAE_LONG_APP_ID' in os.environ:
            return os.environ['GAE_LONG_APP_ID']
        else:
            # on new `env:flex`
            return os.environ.get('GCLOUD_PROJECT')


def get_current_version_name():
    try:
        # works on: GAE proper
        from google.appengine.api import modules
        name = modules.get_current_version_name()
        if not name:
            raise Exception()
        return name
    except:
        if 'GAE_MODULE_VERSION' in os.environ:
            # on `vm:true`
            return os.environ['GAE_MODULE_VERSION']
        else:
            # on new `env:flex` and `env:flex python-compat`
            return os.environ['GAE_VERSION']


def _get_current_version_timestamp_string():
    if 'CURRENT_VERSION_ID' in os.environ:
        # on: GAE proper
        return os.environ['CURRENT_VERSION_ID'].split('.')[-1]
    elif 'GAE_DEPLOYMENT_ID' in os.environ:
        # on `vm:true`, `env:flex` and `env:flex python-compat`
        return os.environ['GAE_DEPLOYMENT_ID']


def get_current_version_id():
    if 'CURRENT_VERSION_ID' in os.environ:
        # on: GAE proper
        return os.environ['CURRENT_VERSION_ID']
    else:
        get_current_version_name() + '.' + _get_current_version_timestamp_string()


def get_default_version_hostname():
    try:
        # on: GAE proper
        from google.appengine.api import app_identity
        hostname = app_identity.get_default_version_hostname()
        if not hostname:
            raise Exception()
        return hostname
    except:
        if 'GAE_APPENGINE_HOSTNAME' in os.environ:
            # on `vm:true` and `env:flex python-compat`
            return os.environ['GAE_APPENGINE_HOSTNAME']
        else:
           # not available on new `env:flex`, so DIY
           return '{}.appspot.com'.format(get_application_id())

[duggelz]

CC @jonparrott @liyanhui1228

[duggelz]

We continue to have discussions about how best to detect/communicate the current environment (GCE, GKE, various flavors of GAE, etc) to application code. However, I can point you to the logic our Cloud client libraries use today:

https://github.com/GoogleCloudPlatform/google-cloud-python/blob/master/logging/google/cloud/logging/client.py#L52

And the logic used by the auth client:

https://github.com/GoogleCloudPlatform/google-auth-library-python/blob/master/google/auth/_default.py#L192

[duggelz]

CC @JustinBeckwith