credential_process doesn't work for federated roles

[m3ldis]

Confirm by changing [ ] to [x] below to ensure that it's a bug:

• [ x ] I've gone though the User Guide and the API reference
• [ x ] I've searched for previous similar issues and didn't find any solution

Describe the bug
It's impossible to use credential_process with federated credentials without manually editing the credentials file every time I want to use the cli.

I use federated roles. Federated roles require constant updating of STS tokens. I have profile [profile user] and federated creds under [user]. I have an internal tool which updates creds for me. It's also a daemon so you run tool& and it will refresh them for you.

If I completely delete [user] in .aws/credentials, the credential_process will execute because the CLI looks for creds in this order:

2020-04-03 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: env
2020-04-03 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: assume-role
2020-04-03 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: assume-role-with-web-identity
2020-04-03 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: shared-credentials-file

If it finds no creds, it will run credential_process. But if it DOES, and those creds are invalid, it just fails and says "invalid creds" instead of running credential_process again.

SDK version number
aws-cli/1.18.36 botocore/1.15.36

Platform/OS/Hardware/Device
What are you running the cli on?
Python/3.7.5 Darwin/18.7.0

To Reproduce (observed behavior)

1. Get temp federated credentials
2. run cli command, eg aws s3 ls
3. invalidate the session token by changing some letters
4. run cli command, eg aws s3 ls

It will fail instead of re-running the credential_process.

Expected behavior
CLI should re-run the credential_process if session tokens are expired or invalid

Logs/output
Get full traceback and error logs by adding --debug to the command.

...
2020-04-03 14:22:24,020 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: env
2020-04-03 14:22:24,020 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: assume-role
2020-04-03 14:22:24,020 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: assume-role-with-web-identity
2020-04-03 14:22:24,020 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: shared-credentials-file
2020-04-03 14:22:24,021 - MainThread - botocore.credentials - INFO - Found credentials in shared credentials file: ~/.aws/credentials
...
2020-04-03 14:22:24,149 - MainThread - botocore.parsers - DEBUG - Response headers: {'x-amz-request-id': '6E9775F8FE088D43', 'x-amz-id-2': '123456789', 'Content-Type': 'application/xml', 'Transfer-Encoding': 'chunked', 'Date': 'Fri, 03 Apr 2020 21:22:23 GMT', 'Connection': 'close', 'Server': 'AmazonS3'}
2020-04-03 14:22:24,149 - MainThread - botocore.parsers - DEBUG - Response body:
b'<?xml version="1.0" encoding="UTF-8"?>\n<Error><Code>InvalidToken</Code><Message>The provided token is malformed or otherwise invalid.</Message><Token-0>IQo...asdfasdfasdfasdf....UE=</Token-0><RequestId>123456789</RequestId><HostId>123456789</HostId></Error>'
2020-04-03 14:22:24,150 - MainThread - botocore.hooks - DEBUG - Event needs-retry.s3.ListBuckets: calling handler <botocore.retryhandler.RetryHandler object at 0x109e2e410>
2020-04-03 14:22:24,150 - MainThread - botocore.retryhandler - DEBUG - No retry needed.
2020-04-03 14:22:24,150 - MainThread - botocore.hooks - DEBUG - Event needs-retry.s3.ListBuckets: calling handler <bound method S3RegionRedirector.redirect_from_error of <botocore.utils.S3RegionRedirector object at 0x109e2e110>>
2020-04-03 14:22:24,150 - MainThread - botocore.hooks - DEBUG - Event after-call.s3.ListBuckets: calling handler <function enhance_error_msg at 0x10b418200>
2020-04-03 14:22:24,152 - MainThread - awscli.clidriver - DEBUG - Exception caught in main()
Traceback (most recent call last):
  File "/user/Library/Python/3.7/lib/python/site-packages/awscli/clidriver.py", line 217, in main
    return command_table[parsed_args.command](remaining, parsed_args)
  File "/user/Library/Python/3.7/lib/python/site-packages/awscli/customizations/commands.py", line 190, in __call__
    parsed_globals)
  File "/user/Library/Python/3.7/lib/python/site-packages/awscli/customizations/commands.py", line 187, in __call__
    return self._run_main(parsed_args, parsed_globals)
  File "/user/Library/Python/3.7/lib/python/site-packages/awscli/customizations/s3/subcommands.py", line 478, in _run_main
    self._list_all_buckets()
  File "/user/Library/Python/3.7/lib/python/site-packages/awscli/customizations/s3/subcommands.py", line 543, in _list_all_buckets
    response_data = self.client.list_buckets()
  File "/user/Library/Python/3.7/lib/python/site-packages/botocore/client.py", line 316, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/user/Library/Python/3.7/lib/python/site-packages/botocore/client.py", line 626, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (InvalidToken) when calling the ListBuckets operation: The provided token is malformed or otherwise invalid.
2020-04-03 14:22:24,153 - MainThread - awscli.clidriver - DEBUG - Exiting with rc 255

An error occurred (InvalidToken) when calling the ListBuckets operation: The provided token is malformed or otherwise invalid.

Additional context
Add any other context about the problem here.

[KaibaLopez]

hi @clhefton ,
thanks for pointing this out, I'll need a bit more help from you because it's hard for me to test this but, could you try adding the profile name and using a region dependent service like:
aws dynamodb list-tables --profile user

I'm expecting this to have the same behavior, but want to be sure before I go further into this.

[m3ldis]

I do get the same error.

Also, I have verified that the script I have for "credential_process" (/usr/local/bin/getcreds) does work if run on its own outside the context of the CLI. So the only issue is re-running it upon finding expired tokens

[stealthycoin]

What is the format of the getcreds output? The process creds expect an expiry time, if the creds are present and not expired the CLI will assume they are valid and try to use them. You didn't mention that they were expired or not, just that they were invalid, in which case the CLI behavior seems correct to me unless I am misunderstanding the scenario.

Can you provide a full sanitized version of your config file with the profiles you metnioned, as well as provide the output format of the process called by the process provider.

[m3ldis]

IMO, it's not "correct" if it doesn't find new creds for you if the ones you have don't work. That's what credential_process is for. But I tested that it doesn't work on legitimately expired creds either; here are some screenshots. It shows last written date to ~/.aws/credentials yesterday, shows creds existing under my profile, and shows the "expired token" error when running a command.

[m3ldis]

As for the getcreds question, it does the following: starts a daemon that updates the ~/.aws/credentials file, sleep(5) to give the daemon time to update the first time, pull the new creds from credentials, output them again in correct JSON format so credential_process doesn't error out.

I.e.,

[profile]
credential_helper=/bin/getcreds
    -> call getcreds, which:
        ->starts my_daemon
        ->reads updated credentials file
        ->outputs correct json based on values above
    -> reads JSON from get_creds (or should)

But it never reaches step 1, which is calling getcreds

[jamessoubry]

Hi, any progress on this issue as I'm getting exactly the same issue using gsts? I need to delete the credentials from the credential file before it updates them. Thanks