Unicode (was Getting Kanji from a .csv file)
pete at lcsql.com
Sun Jun 9 00:41:18 CEST 2013
Thanks Dar, I think I get the picture now. I'll stick with UTF8 for now.
lcSQL Software <http://www.lcsql.com>
On Sat, Jun 8, 2013 at 11:11 AM, Dar Scott <dsc at swcp.com> wrote:
> I encourage you to go to full Unicode.
> That means (for now) using the unicodeText to get text into and out of a
> Then convert that to UTF8 and back for a database with UTF-8 encoding.
> (And at this point you can say that your program only works with UTF-8
> encoding set for SQLite.)
> It is my understanding that SQLite does not have any lossy encodings.
> That is, you don't lose anything by saving to the db. That is, all
> encodings are Unicode. My comments are based on that.
> You probably can't reliably move a db with UTF-16 encoding from one
> machine to another.
> Since your program is general, you probably want to accommodate db with
> UTF-16, UTF-16LE, and UTF-16BE.
> I'm guessing you can store the Unicode you get from the unicodeText
> directly for UTF-16.
> For the others you might have to byte swap.
> Long ago I made a enhancement suggestion to include UTF-16LE and UTF-16BE
> in uniEncode and uniDecode. I don't think it is there, so you will have to
> do it yourself.
> Essentially, you see if what is native for your machine matches your
> target encoding. If not swap. To see if the chars are stored little
> endian or not ...
> Gotta run.
> On Jun 8, 2013, at 11:20 AM, Peter Haworth wrote:
> > I apologize up front for being particularly clueless on this whole
> > character encoding concept. I'm still trying to adjust to speaking
> > American English as opposed to the Queen's English so not too suprising
> > not grasping unicode too well!
> > I understand the concepts and the use of uniencode and unidecode but I
> > don;t understand when I need to care.
> > I'll use my SQLiteAdmin program as an example. It provides schema
> > maintenance and data browsing/update features for SQLite databases and
> > most of the standard LC controls, including datagrids. Users can enter
> > data into it and have it used to INSERT, UPDATE, or DELETE rows. They
> > also type in SELECT criteria and have the qualifying data displayed in
> > field and datagrid controls. Currently, there is no attempt to do any
> > encoding or decoding of data.
> > On my computers here in the USA, I've never had any issues using it on
> > of my databases, but I've never tried to access one whose contents
> > in American English..
> > Now let's say someone in a country whose language requires the use of
> > unicode encoding purchases the program. WIll it work OK for that person
> > terms of entering data into the controls and displaying data in the
> > controls from their database, assuming that the database contains UTF8
> > encoded data? Or do I have to uniencode/decode to ensure things work
> > Now let's say the database is using UTF16 encoding, or anything other
> > UTF8. I can detect that situation in the database and I think I would
> > to use uniencode/decode to deal with it?
> > Now the user takes his UTF8 database and puts it on a colleague's
> > here in the USA with the computer's language settings set to American
> > English. I would then need to decode/encode.... I think.
> >> From the original thread, it seems clear that when I import data into
> > database via SQLiteAdmin, I do need to be aware of the encoding in the
> > imported file and that there may be a way to detect that within the file
> > depending on how it was produced. Conversely, when I export data, I
> > try to create the same marker in the file.
> > And finally, is the simplest way to take care of this to simply
> > uniencode/decode everything using the databases encoding without regard
> > to whether that's necessary or not?
> > Pete
> > lcSQL Software <http://www.lcsql.com>
> > _______________________________________________
> > use-livecode mailing list
> > use-livecode at lists.runrev.com
> > Please visit this url to subscribe, unsubscribe and manage your
> subscription preferences:
> > http://lists.runrev.com/mailman/listinfo/use-livecode
> use-livecode mailing list
> use-livecode at lists.runrev.com
> Please visit this url to subscribe, unsubscribe and manage your
> subscription preferences:
More information about the use-livecode