use-livecode Digest, Vol 224, Issue 9, Re: Decrypting (and encrypting) Large files

[Mark Clark]

Why the preference? It’s not a hard and fast rule by any means, but I suppose because rolling your own means owning your code, flaws and all. In simple system environments I wouldn’t use LC at all, I’d just use shell scripting. Since different users will have different shells (or none) probably best in this case to make it standalone. Calling shell might cause more support cases than a roll your own especially since there are different versions of openssl, LibreSSL v OpenSSL, etc. The default message digest differs from one flavor of openssl to another—looks like built in encrypt is using md5 still? Maybe I’m missing some knob I could turn at encrypt time, wouldn’t be the first time I missed something obvious. 

Yeah, chunking seems to work just fine, although I ran into another issue. I decided to not use cbc (big IV for my use case) and instead use gcm and this seems to be broken somehow in LC 9.6.3. There aren’t any useful error messages that I can see. Just to fast check I downloaded an old stack by Bill Vlahos (from 2012) and checked to see if I was doing something egregiosuly wrong. If I am, I’m not alone;) aes-256-gcm seems to “work” on encrypt and blow up on decrypt. Ditto for 192, 128, etc.

The last option you mentioned doesn’t work in my use case, though it makes plenty of good sense in a standard type data center scenario. 

I really do appreciate all the nice folks on this list. One of the main reasons I enjoy LiveCode is the people who use it.

If I can resolve the use of gcm soonish I’ll do a few more tests at different ranges of blocks. Intuitively it might seem that a larger chunk for encryption would be faster, but that’s not what the tests produced. Below are the results for a 1MB chunk to encrypt vs a 1GB chunk. Obviously the memory requirements for the standalone are much friendlier for the smaller chunk. 

This is from an oldish mac laptop
Processor Name:	Quad-Core Intel Core i7
  Processor Speed:	2.8 GHz
  Number of Processors:	1
  Total Number of Cores:	4
  L2 Cache (per Core):	256 KB
  L3 Cache:	6 MB
  Hyper-Threading Technology:	Enabled
  Memory:	16 GB


cipher is aes-256-cbc

source zip: 5,904,809,647 bytes (5.9 GB on disk)
encrypted zip: 5,904,990,880 bytes (5.92 GB on disk)
decrypted zip: 5,904,809,647 bytes (5.9 GB on disk)
-----
byte range: 1073741824 (1GB)

Total milliseconds required to encrypt in 6 runs: 61884
memory peaked at 4GB, returned to 42 MB after encryption run 

Total milliseconds required to decrypt in 6 runs: 69678
memory peaked at just under 4GB, fluctuates at 2-3.5GB

----

byte range: 1048576 (1 MB)
memory utilization went from 42 MB to 45.5MB
Total milliseconds required to encrypt in 5632 runs: 45918

memory utilization 45.7 MB
Total milliseconds required to decrypt in 5632 runs: 41725

Thanks all, if anyone would like to check out the gcm flavor from a Mac I’d like to know if you get different (better) results. I’m on 11.6.4, btw. 

Mark




> On May 11, 2022, at 11:00 AM, use-livecode-request at lists.runrev.com wrote:
> 
> What is behind the preference to roll your own rather than call existing 
> purpose-built command-line tools from LC with shell()?
> 
> Your chunking described in your latest post seems the way to go, AFAIK 
> pretty much how other tools would handle it.
> 
> Another option:  if you're in an environment where even log files 
> require strong encryption, could you pipe log data to a separate secured 
> log server instead?