[ANN] rsIsValid(Validation functions) library extension for LC8 and a call for contributors

[Roland Huettmann]

Dear Steve

I have been thinking about your offer, but it is not fully clear to me what
to contribute as the field of validation is a huge field in itself. I
assume that your Validation library would mainly target data validation?

As i see it, data validation is usually done on various levels, mainly on
the level of user input (representation layer), and as well on the level of
storing data in a database or elsewhere (mySQL, SQlite, text files, and
others).

*CHARACTER CHECKS*

We make sure that only those characters are entered which are allowed for
the specific field or list.

A numeric field would be constrained to the entry of numeric digits between
0-9, and possibly a minus sign and commas and dots. Depending on the
country of usage, a decimal sign can be a period or a comma. Thousand
separators also come in a variety as commas, high commas (plain
apostrophes), spaces, and dots if you go for international number formats.
The type used locally should somewhere be noted, otherwise a conversation
may lead to unexpected results. It could be marked as a custom property of
a field, and/or the plain number itself (as it is widely accepted with a
decimal point instead of a comma) may be stored as a custom property as
well.

Date representations are even varying much more. The system date gives the
local format, but there should be a representation which all machines on
all platforms allow. For a date, the international representation in the
form YYYY-MM-DD may be the common denominator which then could be
transformed into whatever a database entry may require, or the user likes
to see (system setting).

For a date we would have to check that it is also a valid date (no February
31 for example) and that characters enter are falling within range.

Then hours follow similar rules also being separated according to local
rules, but stored as international accepted HH::MM:SS with or without
leading zeros.

If it is a representation of dates then the maximum entered is 24:00. But
hours, minutes and seconds could be added or subtracted and the output
format could again vary counting in years, months, days, and so on.

There are special cases in other languages and cultural regions.

A field containing an email address follows again certain rules and must
contain one and only one "@" sign. There there is a validity check for
URLs, and if you only like to store US phone numbers, then such numbers
follow a certain pattern. A validity check may be needed.

*LIST CHECKS*

A CVS file may have to checked for validity. Are records missing? Are
delimiters missing? Are there column headers or not? And there are other
formats such as XML and JSON etc. Are they valid? How do we check?

*INTEGRITY CHECKS*

Using hash values the integrity of files or other data sources may be
checked.

*CONSISTENCY CHECKS*

How would we check consistency in data? Did the user enter the required
data in the way the system expects them? Again, difficult to predict and
make it a generic function.
If we had "best practice" official rules going beyond what the IDE script
editor is checking anyway, we might add additional consistency checks for
style, usage of development patterns, etc.

*CROSS-PLATFORM CHECKS*

There could be a way to test automatically if data used on platform A are
the same as on platform B (Windows vs. OSX vs. HTML, vs. Android, vs. iOS
vs. Linux). It should be elaborated what exactly to check and how.

*DATA TYPE CHECKS*

Since in LCS (LiveCode Script) all data is text, it may be difficult to
check better than the already available functions would chech using "if 1
is a number then"... etc. It LCB - not having used it myself - I assume
that the compiler will throw an error when data type consistency is not
maintained and there is no explicit conversation. (But I am not used to it
yet, so I do not know.)

*EXISTENCE CHECKS*

We have to usually check if a file used is actually present, not empty,
etc.The same we do with URLs and databases. Does the connection exist? If
not, what to do?
We also check for windows, or other data being present.

*RANGE CHECKS*

Similar to character checks.... we want to ensure that data is present,
entered, exported or imported being constrained to a certain limit. Data a
d time ranges, number ranges, character ranges,

*SPELLING AND GRAMMAR CHECKS*

For which languages available? a self-learning dictionary attached? Grammar
rules?
Also certain formatting rules could be enforced.

*FORMATTING CHECKS*

For example, the postal address in the USA is differently formatted than
the one in Great Britain or Continental Europe or Japan and China. There
could be a validation process checking for correctness of such national
formatting rules.

Formatting a number, date, etc. (as already mentioned) is different from
region to region.

Email strings, URLs, ... there are formatting rules.

*UNIQUE VALUE CHECKS*

What way do we have to check that a value is unique - and within which
environment. This is not just needed in databases, but may also be applied
elsewhere.

I believe such work should first somehow be specified and its boundaries
also must be defined.

I do not believe that relatively trivial checks (checking for example if
something is a number) are what you are looking for as you and everyone can
do that yourself or ourselves and is often doing it already.

*FRAME WORK ENFORCING CORRECT DATA ENTRY AND REPRESENTATION*

As i could see a more advanced framework, and that would raise my interest
more, is how to enforce data entry and representation for international
customers and an automatic way of converting and storing data the way the
engine or attached databases are using them.

In other words there is a always a representation layer allowing for data
representation and data entry, and then there is a data layers where "raw"
data is stored which is used for any kind of calculation and correct data
storage.

I imagine that it would be great if a user had the possibility to define
properties for a field or list fields (grids) guiding the representation as
well as the storage of data -  and separate both. Domains (or styles) could
also be defined separately which when applied would automatically set the
formatting, enforce validation, and enforce correct storing of data for a
given field, list field, or other container, or attached file/database.

*SUGGESTION*

What would be the perfect solution here? That will vary according to
subjective view, but I could see an enhancement of the Properties and
Behaviour functionality of LiveCode:

In my idea, behaviours including validation checks are defined in a central
place. One could call this "domain" and match such domain concept with
domains as they are known in the database world.

(Small remark:  I do not personally like so much defining a behaviour using
a button. For me a button is something to click on, not to store a
behaviour. I have problems looking at a button as a generic container and
not just as a button )))

Maybe there is a better word than "domain"? Style is not enough as it only
defines some outer visual aspects. But a domain such as a decimal
representation and storage of numbers with or without currency symbols
limiting input of values to what is defined, representing according to
local or system rules, and enforcing data ranges defined by the user,
controlling user input, aligning numbers according to s style template -
once defined it never would have to be defined again. At the same time such
behaviour would allow to store data in the way it is expected in an
associated data storage (database, file, etc.) and such data sources could
also be defined to automatically format data the way such data sources
would expect.

And of course in analogy the same holds true for all kinds of data and
their expected manipulation, representation, input and output, storage and
management - including audio and video, and any kind of files formats, etc.
whether textual or binary.

The user could manage all those settings in one place (not in a single
button please) and be able to assign such properties-behaviours to single
fields or other controls, or to groups of fields and controls. This would
also include using a data grid or other form of list representation. There
will be default settings which can be changed. And the setting for an
individual control will take precedence over the universal settings - as it
is already done using behaviours today.

Personally I really would like to see all such settings, including
behaviours, validation rules, constraints, business rules, visual
representation settings, data source settings in one manageable table where
also all the fields and other controls are defined and can be managed. Some
of this has been done already. My suggestion is just an expansion of the
idea and making it user-friendly. And it includes the validation of data at
least.

I know that this can become very complex.

A big amount of work could then be taken away from the developer and also
casual user (hobbyists, educators, non-programmers) so that more focus is
on the actual task to achieve rather than formatting and validation and
storage concerns.

----

Am I a good-enough LC coder to actually supply scripts here? I think such
work should be structured from your side first so that some people can do
some job here, and other might do another job there. Should it not be a
coordinated effort?

Kind greetings, Roland

P.S. Why not we find a way of validating our own existence? )












On 26 February 2016 at 21:29, Stephen MacLean <smaclean at madmansoft.com>
wrote:

> Hi All,
>
> I'm in process of creating a library extension for LC 8 that contains a
> set of
> functions to provide data validation. I'm basing my initial set of
> functions on
> a subset of the functions provided in ColdFusion, which I use for much of
> my backend development.
>
> This extension will be available for free and under the MIT license as I’m
> trying to avoid any issues with including this in both the community
> edition as well as Indy and Business editions of LC. Source code is
> available on GitHub.
>
> Validation is a very important part of ensuring the data being inputted is
> what is expected and preventing things like SQL injection attacks, etc. We
> all have our own different methods of doing validation and as with most
> things, some methods are better than others.
>
> I'm putting out a call for contributors to provide their solutions to
> different validation functions, for both included and not included in the
> current iteration of rsIsValid. Your’s may be better than mine!
>
> You do not have to know anything about LiveCode Builder, you can just
> submit your current LiveCode Script. I will work to put it into the
> extension via LCB. If you do contribute code, please make sure that any
> needed attribution is included in your code comments.
>
> If you have a validation function you’d like to contribute, you can post
> it in the forum thread linked below or fork it on GitHub. I plan to have
> the initial functions done, packaged and uploaded to the extension area of
> LiveCode by the end of next week. The development branch on GitHub is
> https://github.com/renegadesteve/rsIsValid/tree/development
>
> You can find a list of functions planned for the initial release on the
> forums at: http://forums.livecode.com/viewtopic.php?f=16&t=26653
>
> My hope is that we can together build a very robust library of validation
> tools for the entire LC community.
>
> Best,
>
> Steve MacLean
>
> _______________________________________________
> use-livecode mailing list
> use-livecode at lists.runrev.com
> Please visit this url to subscribe, unsubscribe and manage your
> subscription preferences:
> http://lists.runrev.com/mailman/listinfo/use-livecode