UnicodeDecodeError when accessing request parameters

[bmiklautz]

Hi,

first off thank you for your effort and work on werkzeug and flask they are really great.

In some of our projects where we use flask we regularly see UnicodeDecodeError when accessing request parameters with valid latin1 codes (with request.args.get) . This happens when using other WSGI servers than werkzeug's development server.

As an example the following minimal request will trigger the exception because of the á when gunicorn is used:

index?name=hiá

Received bytes:

b'index?name=hi\xe1'

The request received and passed by the WSGI server sets a valid latin1 encoded str in the environment.

Without handling this directly in werkzeug the only solutions I've found is to
globally catch UnicodeDecodeError or to catch the exception in a try block
(with a wrapper function). Both basically work but are not necessarily what you want.

Therefore I'd like to discuss if there are another solution possible within werkzeug.

I'm happy about any recommendation and hint and gladly create a pull request for a solution.

Steps to reproduce

To reproduce simply create a venv and install the required packages:

python3 -m venv venv
. ./venv/bin/activate
pip install --upgrade pip
pip install werkzeug gunicorn

For running the commands it is assumed that the venv is activated.

As most libraries and tools automatically do percent encoding for characters > 128 or escape certain chars a raw request is used:

#/usr/bin/env python3
# file called raw_request.py
import socket

HOST = '127.0.0.1'
hostname = 'test.localhost'
PORT = 5000

with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as s:
        s.connect((HOST, PORT))
        s.sendall(b'GET /index?name=hi'+'á'.encode('latin1') + b' HTTP/1.1\r\n')
        s.sendall(b"Host: "+hostname.encode('utf-8')+b"\r\n")
        s.sendall(b"User-Agent: raw-request/0.0.1\r\n")
        s.sendall(b"\r\n")
        data = s.recv(2048)

print(f"Response {data!r}")

Server and WSGI app - example was taken from https://werkzeug.palletsprojects.com/en/3.0.x/tutorial/

# file called test.py
from werkzeug.wrappers import Request, Response

def application(environ, start_response):
    request = Request(environ)
    text = f"Hello {request.args.get('name', 'World')}!"
    response = Response(text, mimetype='text/plain')
    return response(environ, start_response)


if __name__ == '__main__':
    from werkzeug.serving import run_simple
    app = application
    run_simple('127.0.0.1', 5000, app, use_debugger=True, use_reloader=True)

Start gunicorn:

gunicorn 'test:application' --bind 127.0.0.1:5000

If you run the python raw_request.py script against gunicorn you will see the UnicodeDecodeError

Same run of python raw_request.py against the development server started with

python test.py

everything is fine.

Details

To find out the difference in parameter handling between werkzeug's and other UWSGI servers
I tracked the path of the parameters from reception to access. As example for an other
server I've used gunicorn.

werkzeugs development server

0.) byte are received and converted to string by http/server str(self.raw_requestline, 'iso-8859-1')
1.) string to byte to string: encode('utf-8').decode("latin1") in werkzeug/wsgi.py
2.) string to byte encode("latin1") werkzeug/wrappers/request.py (Request init)
3.) byte to string decode("utf-8") werkzeug/sansio/request.py args

All operations applied to a received parameter would be:

str(b'hi\xe1', 'iso-8859-1').encode('utf-8').decode("latin1").encode("latin1").decode('utf-8')

gunicorn

0.) str(b, 'latin1') received by the respective worker (gunicorn/http/message.py parse_request_line -> urlsplit in werkzeug/utils)
2.) string to byte encode("latin1") werkzeug/wrappers/request.py (Request init)
3.) byte to string decode("utf-8") werkzeug/sansio/request.py args

All operations applied to a received parameter would be:

str(b'hi\xe1', 'latin1').encode('latin1').decode('utf-8')
    

In 0.) the request is received and initially decoded to latin1/iso-8859-1. 2.) and 3.) is done in werkzeug regardless of the WSGI server used.

The difference is in 1.) the encode('utf-8').decode("latin1"). This is only done when the development server is used.

Possible fixes
Setting the error handler of decode to ignore "fixes" the issue but might remove valid codes.

diff --git src/werkzeug/sansio/request.py src/werkzeug/sansio/request.py
index dd0805d7..e6cdf13b 100644
--- src/werkzeug/sansio/request.py
+++ src/werkzeug/sansio/request.py
@@ -172,7 +172,7 @@ class Request:
         """
         return self.parameter_storage_class(
             parse_qsl(
-                self.query_string.decode(),
+                self.query_string.decode(errors='ignore'),
                 keep_blank_values=True,
                 errors="werkzeug.url_quote", 
             )        

Another possibility would be to do the encode('utf-8').decode("latin1") in werkzeug/wrappers/request.py or werkzeug/sansio/request.py instead of werkzeug/wsgi.py then handling would be the same for the development server as well for any other WSGI server.

If you need any further information or have questions please let me know.

Thank you for taking time!

[davidism]

From the URL spec: https://url.spec.whatwg.org/#percent-encoded-bytes, emphasis added.

The C0 control percent-encode set are the C0 controls and all code points greater than U+007E (~).

The query percent-encode set is the C0 control percent-encode set and U+0020 SPACE, U+0022 ("), U+0023 (#), U+003C (<), and U+003E (>).

You said:

As most libraries and tools automatically do percent encoding for characters > 128 or escape certain chars a raw request is used

Therefore, you're deliberately sending data against spec and getting an unexpected result. You need to percent encode the characters that must be encoded.

[bmiklautz]

@davidism thank you for your answer. These requests are nothing we send ourselves but rather requests we receive in the wild. I was maybe not clear enough on this in my description. The raw_request.py is just an example and the simplest way to demonstrate and reproduce the issue :).

I'm absolutely agree there shouldn't be any request with code points greater than U+007E (~) without percent-encoding however there are and as recipient one can't really influence them.
When werkzeug's development server is used these cases are handled and work fine. It would be really nice to have the same handling for other WSGI servers at well or at least a possibility to decide if an exception should be thrown. Invalid encoded parameters, or parts, could also be ignored or returned as empty string.