UnicodeEncodeError：'latin-1' codec can't encode characters in position 0-1: ordinal not in range(256)

Question

i am a newer in python.Today when I write some search function I met an error.well, I use sqlalchemy orm to do that, in my function,I input a chinese word as the key word.The html page give me an UnicodeEncodeError at /user/search:'latin-1' codec can't encode characters in position 0-1: ordinal not in range(256). and my code is like this:

def user_search(request):
    name = request.GET.get('name').strip()
    user_list = list()

    if name:
        user_list = User.get_by_name(name)

class User(object):
    @classmethod
    def get_by_name(cls, name):
        return DBSession.query(cls).filter(cls.name==name)

and the Traceback is here:

Traceback:
    File "/usr/local/lib/python2.6/dist-packages/django/core/handlers/base.py" in get_response
  111.                         response = callback(request, *callback_args, **callback_kwargs)
File "/home/jiankong/git/admin-server/lib/decorators.py" in wrapper
  75.                     return func(request, *args, **kwargs)
File "/home/jiankong/git/admin-server/lib/decorators.py" in wrapper
  39.             output = function(request, *args, **kwargs)
File "/home/jiankong/git/admin-server/apps/user/user_views.py" in user_search
  47.     users = jump_page(paginator, page)
File "/home/jiankong/git/admin-server/apps/user/utils.py" in jump_page
  92.         return paginator.page(1)
File "/usr/local/lib/python2.6/dist-packages/django/core/paginator.py" in page
  37.         number = self.validate_number(number)
File "/usr/local/lib/python2.6/dist-packages/django/core/paginator.py" in validate_number
  28.         if number > self.num_pages:
File "/usr/local/lib/python2.6/dist-packages/django/core/paginator.py" in _get_num_pages
  60.             if self.count == 0 and not self.allow_empty_first_page:
File "/usr/local/lib/python2.6/dist-packages/django/core/paginator.py" in _get_count
  48.                 self._count = self.object_list.count()
File "/usr/local/lib/python2.6/dist-packages/SQLAlchemy-0.8.1dev-py2.6-linux-i686.egg/sqlalchemy/orm/query.py" in count
  2414.         return self.from_self(col).scalar()
File "/usr/local/lib/python2.6/dist-packages/SQLAlchemy-0.8.1dev-py2.6-linux-i686.egg/sqlalchemy/orm/query.py" in scalar
  2240.             ret = self.one()
File "/usr/local/lib/python2.6/dist-packages/SQLAlchemy-0.8.1dev-py2.6-linux-i686.egg/sqlalchemy/orm/query.py" in one
  2209.         ret = list(self)
File "/usr/local/lib/python2.6/dist-packages/SQLAlchemy-0.8.1dev-py2.6-linux-i686.egg/sqlalchemy/orm/query.py" in __iter__
  2252.         return self._execute_and_instances(context)
File "/usr/local/lib/python2.6/dist-packages/SQLAlchemy-0.8.1dev-py2.6-linux-i686.egg/sqlalchemy/orm/query.py" in _execute_and_instances
  2267.         result = conn.execute(querycontext.statement, self._params)
File "/usr/local/lib/python2.6/dist-packages/SQLAlchemy-0.8.1dev-py2.6-linux-i686.egg/sqlalchemy/engine/base.py" in execute
  664.                                                 params)
File "/usr/local/lib/python2.6/dist-packages/SQLAlchemy-0.8.1dev-py2.6-linux-i686.egg/sqlalchemy/engine/base.py" in _execute_clauseelement
  764.             compiled_sql, distilled_params
File "/usr/local/lib/python2.6/dist-packages/SQLAlchemy-0.8.1dev-py2.6-linux-i686.egg/sqlalchemy/engine/base.py" in _execute_context
  871.                                     context)
File "/usr/local/lib/python2.6/dist-packages/SQLAlchemy-0.8.1dev-py2.6-linux-i686.egg/sqlalchemy/engine/default.py" in do_execute
  324.         cursor.execute(statement, parameters)
File "/usr/local/lib/python2.6/dist-packages/MySQL_python-1.2.4-py2.6-linux-i686.egg/MySQLdb/cursors.py" in execute
  183.             query = query % db.literal(args)
File "/usr/local/lib/python2.6/dist-packages/MySQL_python-1.2.4-py2.6-linux-i686.egg/MySQLdb/connections.py" in literal
  264.         return self.escape(o, self.encoders)
File "/usr/local/lib/python2.6/dist-packages/MySQL_python-1.2.4-py2.6-linux-i686.egg/MySQLdb/connections.py" in unicode_literal
  202.                 return db.literal(u.encode(unicode_literal.charset))

Exception Type: UnicodeEncodeError at /user/search
Exception Value: 'latin-1' codec can't encode characters in position 0-1: ordinal not in range(256)`

when I met the error, I did a test in python shell, it worked well,the code is here:

from apps.user.models import User user = User.get_by_name('某人').first() print user print user.name 某人

so what can I do to let it worked in my html page?much appreciate!!

Mondragon

I guess type of User.name is String? Try to change it to Unicode. – Commonable

Answer 1

I'm assuming that you're using MySQL with the MySQLdb driver here.

The default encoding used by the MySQLdb driver is latin-1, which does not support your character set. You'll need to use UTF-8 (or others, but UTF-8 is the most common) to be able to communicate with your database through MySQLdb (see http://docs.sqlalchemy.org/en/rel_0_8/dialects/mysql.html#unicode).

To do such a thing, create your engine with the following line:

create_engine('mysql+mysqldb://USER:@SERVER:PORT/DB?charset=utf8', encoding='utf-8')

You can also construct your engine url using the sqlalchemy.engine.url.URL class, and send it to the create engine function. I find it useful when you have your settings in a config file.

import sqlalchemy.engine.url as url

engine_url = url.URL(
    drivername='mysql+' + cfg['MYSQL_PYTHON_DRIVER'],
    host=cfg['MYSQL_HOST'],
    port=cfg['MYSQL_PORT'],
    username=cfg['MYSQL_USER'],
    password=cfg['MYSQL_PWD'],
    database=cfg['MYSQL_DB'],
    query={'charset': 'utf8'}
)
db = create_engine(engine_url, encoding='utf-8')

Hope that helps.

Indicia

Answer 2

based on your stacktrace, you're using MySQL Python with unicode encoding turned on, since it's doing an encode. So you likely need to specify a comaptible encoding (note this is all settings used by the MySQLdb DBAPI, SQLalhcemy just passes them through):

create_engine('mysql+mysqldb:///mydb?charset=utf8&use_unicode=1')

http://docs.sqlalchemy.org/en/rel_0_8/dialects/mysql.html#unicode

Stearic