I am hitting an intermittent issue with a lambda taking binary inputs where AWS API gateway throws a 500 error before my lambda gets run.
X-Ray did nothing for me. The only relevant information is if you enable detailed logging:
Execution failed due to configuration error: Unable to transform request
Various mentions/discussion of this:
https://www.reddit.com/r/aws/comments/5djiqw/binary_data_now_supported_by_api_gateway/
https://stackoverflow.com/questions/55667066/aws-api-gateway-error-while-making-options-request
https://stackoverflow.com/questions/48094698/unable-to-transform-request-to-binary
https://github.com/aws/serverless-application-model/issues/553
The final resource above is telling in that PRs were not merged.
Initially comparing the output between a working endpoint and non working one showed a big difference:
aws apigateway --region us-east-1 get-integration --rest-api-id <ID> --resource-id <ID> --http-method POST
I redeployed the non working one with async set to false, and the two command outputs from just above now matched, but it did not seem to fix the issue, but when I waited some time and ran the test again it succeeded. Obviously not ideal, because we need deployment to always work without failure.
Re-deployed with async set to true and the API did not start failing. That seems unlikely, so deleting and recreating with async set to true.
New theory - a new deployment with async set to true will always fail