HTTP Get - Empty response body

[jesuslg123]

Hello,

I'm having a really weird issue where the response body of a GET http request is empty but it should have a JSON object.

The request is done to a GoPro camera, using the Wifi API, and it's actually works, I can see my GoPro changing the state and the request response is 200, but the body is empty.

I have been doing some testing and I found a pretty weird behaviour:

• Any request to the GoPro API is getting empty body response using UIFlow/Microphyton (while body has content doing the request from my computer).
• Serving a GoPro JSON object from my computer it works fine on UIFlow/Microphyton, response has content.
• Any requests to my GoPro using Arduino IDE with WifiClient lib is getting the valid json response.

I'm totally stuck here, not sure if M5Stack firmware or urequest lib has some kind of bug or whats it is happening. Any help is more than welcome :)

Thanks!

** Solution? **

[robalstona]

Maybe you need set a some parameters in header of your request. like "Content-type". I use Restler app on android to testing requests.

[jesuslg123]

Thank you for your answer, after all day testing I found the issue:

The urequest library, which is used for the firmware, has an implementation that fails with the GoPro requests due to how it cleans the request body.

During the clean up process of the body its split content by if not l or l == b"\r\n":,
which probably is ok for most of the requests, but not too flexible or safe, but the GoPro use \n\n to split the Content-Type line from the actual body response.

So in order to fix the issue, I need to modify the urequests implementation, from:
if not l or l == b"\r\n":
to
if not l or l == b"\n" or l == b"\r\n":
with this change, everything works fine.

So my question is, it is possible to replace the urequests library included in the firmware with my fork/custom changes? I have seen how to load external libs, but I don't know the path for the urequest lib.

Thank you!

[jesuslg123]

I can not edit the post, sorry.

Sample of the response bodies:

Most common and working

See its use all the time \r\n

GoPro sample, and not working

See it ends with \n\n b'HTTP/1.0 200 OK\r\n

[robalstona]

I use urequests in raw python and i use something like this code below to parse response (i delete some unwanted characters by replacing it with empty string)

response = urequests.get('http://...')

if response is a json

data = json.loads(response.text.replace('STRING_TO_REPLACE', 'NEW_STRING')

other text

data = response.text.replace('STRING_TO_REPLACE', 'NEW_STRING')

[jesuslg123]

@robalstona said in HTTP Get - Empty response body:

I use urequests in raw python and i use something like this code below to parse response (i delete some unwanted characters by replacing it with empty string)

response = urequests.get('http://...')

if response is a json

data = json.loads(response.text.replace('STRING_TO_REPLACE', 'NEW_STRING')

other text

data = response.text.replace('STRING_TO_REPLACE', 'NEW_STRING')

The problem is that the library is breaking the response before return it to me. When doing
esponse = urequests.get('http://...')
at that point the response body is already broken, that why I need to customize the library. It in deeper level the issue.

[jesuslg123]

OLD ANSWER

So, I could workaround this issue after some work and before close and forget this thread here forever, let me share it:

The problem:

• GoPro wifi API is not fully following the HTTP protocol specifications.
  • Headers and body content is separated by \n instead of \r\n
• uRequests library is very strict on response parsing, while mostly any other libraries/browser/etc handle requests with some "malformed" bodies, uRequests does not.

Failed attempt:
- Upload and override the uRequests library include on the Stick5C firmware.
- Continue with normality

The solution:

• Stop using UIFlow IDE 😢
• Upload to the Stick5C the modified version of uRequests
• Use Visual Studio code or any other IDE to continue the development.
• Override the uRequests library usage with a new import on the code.

I hope this is useful to someone else, any suggestion is also more than welcome.

Thanks :)

[jesuslg123]

Just finished to write it down all this and I think I the failed attempt it could work using the execute code block to make the import override and keep using the UIFlow IDE 😱 That this makes sense?

New attempt

• Upload to the Stick5C the modified version of uRequests
• Keep using UIFlow IDE 🎉
• Override the uRequests library usage with a new import on the code using execute code block.

I will try it and update you.

[jesuslg123]

So, I could workaround this issue after some work and before close and forget this thread here forever, let me share it:

The problem:

• GoPro wifi API is not fully following the HTTP protocol specifications.
  • Headers and body content is separated by \n instead of \r\n
• uRequests library is very strict on response parsing, while mostly any other libraries/browser/etc handle requests with some "malformed" bodies, uRequests does not.

UIFlow IDE solution:

• Upload to the Stick5C the modified version of uRequests
  
• Override the uRequests library usage with a new import using execute code block.
  
• From that moment on (only on this app) it will use the modified library.

No UIFlow IDE solution:

• Stop using UIFlow IDE 😢
• Upload to the Stick5C the modified version of uRequests
• Use Visual Studio code or any other IDE to continue the development.
• Override the uRequests library usage with a new import on the code.

I hope this is useful to someone else, any suggestion is also more than welcome.

Thanks 😄