r/Python • u/ikkebr • Mar 07 '17
CIA uses Python (a lot)
From the latest Vault 7 leaks from Wikileaks, we can see that CIA uses a lot of Python in its secret hacking tools.
Most notably in the Assassin, Caterpillar, MagicViking and Hornet projects.
Unfortunatelly, no files from these projects have been released yet. But if you look at the dump that was released, there are plenty of .py files, and even PIL is included.
There are even Coding Conventions: https://wikileaks.org/ciav7p1/cms/page_26607631.html
You can see more Python-related stuff here: https://search.wikileaks.org/?query=python&exact_phrase=&any_of=&exclude_words=&document_date_start=&document_date_end=&released_date_start=&released_date_end=&publication_type%5B%5D=51&new_search=False&order_by=most_relevant#results
9
393
u/Exodus111 Mar 07 '17 edited Mar 08 '17
Global Variables
Avoid global variables.
Thank you CIA.
49
15
13
u/nick_t1000 aiohttp Mar 08 '17
But the CIA is supposed to only operate globally and overseas, not locally. CIA SPYING DOMESTICALLY CONFIRMED
6
u/Dualblade20 Mar 08 '17
Everyone knows this, though. Why is the CIA hiring people who don't know how to avoid muddling global scope? Maybe it's just a guide and has things like that for completeness, but still it sort of bothers me.
28
u/Exodus111 Mar 08 '17
Think of it like this, the CIA pays you by the hour to type out a style guide, you might as well include everything.
16
u/MereInterest Mar 08 '17
Believe me, not everybody knows this. These are arguments I've heard in favor of using global variables.
But I don't want to pass the configuration around to everything that needs it.
It's simpler and easier to reason about.
I can adjust the value in one place and change the behavior everywhere in the program.
We just have to put locks around it, and it will be threadsafe.
I can adjust the behavior of something deep within the call stack without needing to change the function signature all the way down.
Now, I know what you're thinking. These aren't actually arguments in favor of global variables. These are many things that are horribly wrong with global variables. That this makes it impossible to reason about code without knowing the runtime state of the problem.
At some point, it is just easier to have a style guide, then use an argument from authority.
1
u/wewbull Mar 08 '17
This is actually my problem with OOP too. What are classes but smaller, not quite as global, scopes? OOP doesn't solve the issues, just pushes them down a level.
3
u/MereInterest Mar 09 '17
It really depends on how you are using classes. If you have some God-object that owns everything, and does all of its work directly within the methods, then yes, it is similar to a global scope. Those are their own antipatterns.
Classes provide a way to have limited persistency. For example, a generator that yields each Fibonacci number in sequence requires some state. You can (a) store the state in global space, (b) require the user to maintain the state, or (c) have an object to manage the state. (a) is error-prone, (b) is irritating, but (c) is very useful.
Unlike global variables, classes can have access restrictions. This means that you don't have to look through the entire codebase for changes, just the class itself. Unlike global variables, classes have a limited lifetime. That means you don't need to know the entire runtime state of the program, just the runtime state of callers into the class. If your classes are acting and looking like global variables, I would say that is an issue on its own.
Where I will agree is that not everything needs to be a class. If a calculation does not require any persistent state, then it should be implemented as a pure function.
4
u/bonestormII May 27 '17
I like the reasoning you provide here. I've read plenty of material about classes in python, but it seems slightly uncommon to me to hear someone just plainly say, "Use classes when you require some degree of state." That is a true and useful statement.
That said, I don't totally follow what you mean when you say
Unlike global variables, classes have a limited lifetime. That means you don't need to know the entire runtime state of the program, just the runtime state of callers into the class.
Aren't both just treated by objects subject to the ref counting of the garbage collector? When you refer to the lifetime of the object, that is what I think of. It seems like you are referring to the compartmentalized scope of a class, more than the lifetime of the object.
It's also relevant to note that in python, classes are also generally useful for any situation in which you want to customize the behavior of the object itself (via inheritance, dunder methods, metaclasses, etc.), regardless of whether state is a factor.
To be honest, almost any example I try to imagine in which you would care about that kind of control would involve some degree of state :P ... but the distinction seems relevant to note.
1
u/MereInterest May 27 '17
Good point, I wasn't entirely clear about my point on the lifetimes, and good catch, that I am somewhat conflating scope and lifetime. This is partly a habit that I picked up from C++, where a variable's lifetime is primarily determined by its scope, and unlike python, cannot be extended beyond the scope that owns it.
In python, you are absolutely correct that local variables and global variables have the same reference counting scheme to determine their lifetime. That said, there is still a connection between scope and lifetime. An object's lifetime is the maximum of the lifetimes of any scopes that contain that object. Since the global scope starts at program initialization and ends at program close, any variable in the global scope must have an indefinite lifetime.
I think that, in terms of global variables, it is both the lifetime and the scope that are issues. The unlimited scope means that you need to examine all code in order to know what could modify the variable. The unlimited lifetime means that you need to watch the running of the program from start to finish, and that the behavior may not be captured in any smaller test.
You are also correct that someone may want to have some behavior customization without needing to have any state in the object itself, though I'm having difficulty as well thinking of any such cases. Perhaps if there were a series of related callbacks, all of would be implemented as methods on a single class, though that would be a rather unusual situation, and would probably be better served with a namespace or dictionary.
1
u/glacierre2 Mar 09 '17
I do OOP, and maintain several serious libs at work.
But I also do some reference implementations in python (to later be translated to C++ in a uC, for example). And then I go full retard on the globals, because I find the clear reading of the example code and the clean functions that get just one/two parameters are more important than robustness.
Of course I still follow two rules:
That kind of code is a reference, NEVER production
Even then, the globals are to be read in multiple places, like a C #define, NEVER to be altered of used to pass information back an forth.
2
u/MereInterest Mar 09 '17
Absolutely agreed. Reference implementations are one thing, while production implementations are completely different. One for testing out new features, quickly hacking in something that might be gone in 5 minutes, while the other is for reliability.
I would add to your second rule. The value of the globals should be determined solely from the source code. That is, using a
#definerather thanstatic const val = read_from_config_file();I work with a framework that does the latter, and it can result in bugs that only ever appear on one person's machine.-1
126
u/dr_g89 Mar 07 '17
If you read through some of the files there are some cool libraries they are referencing. Has anyone used angr before?
57
u/tunisia3507 Mar 07 '17
I've read the readme three times and still don't know what it's for.
97
u/dr_g89 Mar 07 '17
By reading this, you'll become an angr pro and will be able to fold binaries to your whim
If i understand what I've read correctly this is a tool for picking apart and experimenting with existing compiled binaries. My guess is to probe for exploits.
61
u/david5813 Mar 07 '17
It is. If I remember correctly the Shellphish team has given a talk at either Defcon or BlackHat about this. A part of their research is being able to perform a completely automated CTF where the systems identify vulnerabilities and even patch them on the fly.
It is a really cool topic and when I get back to a position to be able to search for it I will happily post a link a talk about it if no one else has already.
Edit: https://www.youtube.com/watch?v=oznsT-ptAbk
You can find more info on their site. http://angr.io/
16
u/hugthemachines Mar 07 '17
Tfw someone told you to do a talk for a certain time and you only have half that and fill 50% with "aaaaaa" ;-)
2
u/flutefreak7 Mar 13 '17
Besides exploits there's also the intelligence aspect of trying to figure out what a binary does. Say you recovered a performance model for a weapon, radar, communications system, etc, if all you have is the binary it could be hard to get it working, but tools like this would help.
8
u/sillycyco Mar 08 '17
I've read the readme three times and still don't know what it's for.
angr is similar to debugging tools you may have used, but entirely within python. Full decomposition of logic, and allowing for analysis of code at a much lower level. Aka scan for exploitable functionality in existing binary code, among other things. Pretty slick actually.
1
u/Dababolical Mar 09 '17 edited Mar 09 '17
I know very little about this, but coincidentally read about IDA today. https://www.hex-rays.com/index.shtml
Is this the python equivalent of the suite (I'm sure there are some differences).
Would someone wanting to implement DRM on their application possibly use this library to make sure their DRM is secure after the application is compiled? I am just trying to think of applications for this library, because it is really interesting.
1
u/sillycyco Mar 09 '17
Ya it is similar to hex-rays, have a look at this talk at Defcon for its various uses: https://www.youtube.com/watch?v=oznsT-ptAbk
You could use it to do analysis of your binaries, attempt to see if you can analyze and get around whatever it is you are trying to protect.
17
u/cdrootrmdashrfstar Mar 08 '17 edited Mar 08 '17
I'm currently enrolled in an undergrad practical fundamentals of cybersecurity course at my uni, and last week, we used angr to script what we'd normally use gdb for in what's called a "capture the flag" assignment.
The creator of the binary will hide away a specific string, like "flag{th1s_1$_th3_fl4g}", purposefully deep within the binary and only obtainable (normally) through some modification to the execution order of the binary.
Originally, we learned to use the GNU debugger GDB for many weeks of CTF assignments, but last week we used Angr to script a way to explore many, many paths of execution in a program. Through adding constraints to inputs and explicting marking some parts of the binary as "avoid", angr would eventually navigate it's way to the end of execution (and hopefully result in a flag!) through some sort of either statistical model (maybe similar to sqlmap?) or through simply brute forcing all possible routes of execution. It's a very complex tool with lots of depth, and I've only really scratched the surface.
We might be doing more of this in the later weeks, but this week moving onto forensics, buffer overflow attacks, shellcode, and circumventing stack protections by compilers. In relation to python, we're using a library called pwntools (in addition to the obvious and incredibly useful set of tools provided by Linux and GNU).
7
u/dr_g89 Mar 08 '17
I'm going to look into pwntools. Where do you go to Uni if you don't mind me asking
7
u/cdrootrmdashrfstar Mar 08 '17
I'm a sophomore at a state school in the southeast US. It's a decent school (it sure has taught me a whole lot in a short amount of time), and we have a great offensive cybersecurity program which very often leads to jobs in places like the CIA, NSA, or the FBI.
3
u/dr_g89 Mar 08 '17
Super interesting man. I run the dev side of a software firm out in LA and have always been way to busy to spend too much time looking into this stuff. If you have a spare second I'd love to know what books on the this stuff they are having you read at school.
13
u/cdrootrmdashrfstar Mar 08 '17 edited Mar 09 '17
Recommended reading: “Hacking: The Art of Exploitation, 2nd Edition” by Jon Erickson: this is a book with accurate and detailed descriptions and commands of common vulnerabilities and corresponding exploits. It is an excellent book for understanding buffer overflow vulnerabilities, string format vulnerabilities, and shellcode, and other exploitation development.
“The Web Application Hacker's Handbook: Finding and Exploiting Security Flaws” by Dafydd Stuttard and Marcus Pinto. The book provides a comprehensive and thorough coverage of web security mechanisms, and web vulnerabilities.
“Information Security,” 2nd Edition, (ISBN 978-0-470-62639-9), Wiley, 2011, by Mark Stamp. The book provides a good coverage on commonly used cryptographic algorithms and cryptanalysis techniques, and security protocols.
Edit: Thank you so much for the gold! It's very much appreciated.
3
u/dr_g89 Mar 08 '17
Dude thanks a ton, especially on the lectures / course stuff, super interesting!
2
u/cdrootrmdashrfstar Mar 08 '17
Absolutely, let me know if you have any questions.
→ More replies (11)1
1
u/registered_tosaythis Mar 11 '17
I've had that first hacking book for like 2 years and haven't spent enough time to finish it!
2
u/cdrootrmdashrfstar Mar 08 '17
Here is our course calendar page containing homework assignments and presentations covering the information we learn in lecture.
11
Mar 08 '17
Christ I hope whoever wrote this library puts it in his CV. "Developed a library used by the CIA"
5
4
u/dr_g89 Mar 08 '17
Some of the other comments on this thread led me to discover it came out of the cybersecurity lab at UCSB. One of the authors is apparently a local high schooler from their community.
3
u/ifatree Mar 08 '17
interesting. are DART, UNDERMINE, PALANTIR, or TYBASE familiar names to any of you guys?
2
u/ssiwhw Mar 08 '17
3
u/HelperBot_ Mar 08 '17
Non-Mobile link: https://en.wikipedia.org/wiki/Palantir_Technologies
HelperBot v1.1 /r/HelperBot_ I am a bot. Please message /u/swim1929 with any feedback and/or hate. Counter: 40912
0
Mar 07 '17
[deleted]
13
u/ebilgenius Mar 07 '17
Took me a minute
3
u/jarins Mar 08 '17
Still don't get it.
42
u/Jesus_Harold_Christ Mar 08 '17
If your username was "I_DONT_NOTICE_USERNAMES" this thread would have been perfect
0
30
u/rockyrainy Mar 08 '17
Parentheses
Use parentheses sparingly.
/r/lisp is gonna have a heart attack.
23
96
u/novel_yet_trivial Mar 07 '17
I'd be a lot more surprised if they didn't.
15
Mar 07 '17
Really? There are plenty of good reasons to use Python, but there are plenty of other good tools out there too with more or less equal reasons to use them.
17
u/zynix Cpt. Code Monkey & Internet of tomorrow Mar 07 '17
Quicker to iterate and then optimize with c/c++ as needed.
6
Mar 08 '17
Sure, and that's a valid and common use of Python. I was just being pedantic and challenging the statement "I'd be surprised if they're not using it."
You can iterate quickly with dozens of languages and then optimize with C or C++. Python is a great choice, but not so unquestionably ahead of all possible competition that it would be surprising if they picked something else.
7
u/Ph0X Mar 08 '17
Yep, check out any CTF event, it's full of Python. It's the perfect tool for trying a lot of crazy ideas very rapidly and prototype.
6
u/JackOhBlades Mar 08 '17 edited Mar 08 '17
False. I iterate fast using C89 and optimise with Brainfuck as needed. ;)
66
u/novel_yet_trivial Mar 07 '17
I agree. I'd also be surprised if they didn't use java, c++, bash, etc...
This is a professional organization and I'd expect them to use the proper tool for every job.
176
Mar 07 '17
This is a professional organization and I'd expect them to use the proper tool for every job.
import waterboarding
183
u/JohnLocksTheKey Mar 07 '17 edited Mar 08 '17
ImportError: No module named waterboarding.
# damn it! I always forget...
from Warcrimes import waterboarding
...
69
21
u/stevarino Mar 07 '17
Unless you're using Python 3, in which case you need to specify the war_crimes library.
9
u/avball Mar 08 '17
PEP-8 would like a space after your hash.
6
u/JohnLocksTheKey Mar 08 '17 edited Mar 08 '17
That was actually bugging me all day!! Fine, I'll correct it...
EDIT: I was hoping no one would notice
→ More replies (5)1
5
0
Mar 07 '17
[deleted]
18
u/brtt3000 Mar 07 '17
Guantanamo started as a PHP hacker camp.
7
u/SchwarzerKaffee Mar 08 '17
The UN has declared that forcing people to code in PHP is a form of torture.
3
u/Ph0X Mar 08 '17
"proper", somehow Python isn't proper?
If you go to any CTF event, you'll find Python everywhere. Exploring binaries and testing exploits is a very interactive process, which is why Python is the perfect fit. You want to try a lot of different approaches and prototype very rapidly.
You don't want to spend an hour just to see what you made is useless. Python is the perfect tool for quickly trying crazy ideas and see what works.
1
-12
4
u/desmoulinmichel Mar 08 '17
Yeah but Python is kinda the default tool in forensic and penetration testing.
2
Mar 08 '17
Good point. I had assumed the CIA would prefer to make and use their own tools. But it makes as much sense for them to take open source tools and improve on them as it does for anyone else.
-1
12
u/wolf2600 Mar 07 '17
The best 10x hacking tools are all written in Rust.
3
u/Dillinur Mar 07 '17
Care to share any link?
31
u/wolf2600 Mar 07 '17 edited Mar 07 '17
No, because it was a joke. Maybe I should have said 'Haskell' instead.
1
1
1
u/sillycyco Mar 08 '17
Maybe I should have said 'Haskell' instead.
Actually Smalltalk via Lisp would be far more preferable.
0
Mar 08 '17
[deleted]
0
Mar 08 '17
[deleted]
5
Mar 08 '17
DevNull as a service
3
48
u/Revik Mar 07 '17
Everyone in IT security uses Python.
38
u/alcalde Mar 08 '17
Everyone everywhere uses Python. :-)
7
u/sourcecodesurgeon Mar 08 '17
My team doesn't use Python, only Java and Ruby :( (though Ruby has slowly started to grow on me over the last year)
41
u/AZNman1111 Mar 08 '17
)
Sorry I think you dropped this
→ More replies (10)18
1
151
u/lovestowritecode Mar 08 '17
If Trump read the README he would see "Avoid global variables" and think "glad the CIA using American variables"
193
u/sweetbeems Mar 08 '17
he wouldn't like all the imports though
8
6
3
10
2
u/snake_case-kebab-cas Mar 08 '17
We will no longer surrender our code to the false song of global variables
0
u/Naihonn Mar 08 '17
Oh well, if CIA had used Python3 exclusively there would be no Trump now... Because we all know how bad is Python2 at handling non ASCII, and I will for no obvious reason at all mention for example, text in azbuka. :0)
18
u/vipul20 Mar 08 '17
I like the parting words in the coding conventions page:
Use common sense and BE CONSISTENT. If you are editing code, take a few minutes to look at the code around you and determine its style. If their comments have little boxes of stars around them, make your comments have little boxes of stars around them too.
5
5
u/XtremeGoose f'I only use Py {sys.version[:3]}' Mar 07 '17
Are the .py files here encrypted or something?
8
0
u/britm0b Mar 08 '17 edited Mar 08 '17
I'm prettt sure Wikileaks isn't letting the hacks wreck havoc yet, they've contacted the companies that have hack-able products, most likely to give them the exploit, so said company can patch it.Apparently not.
3
u/Ph0X Mar 08 '17
They're PDF files. It loads as PDF in chrome for me when I click them (even if they have .py extension).
1
u/britm0b Mar 08 '17
So you can view the code? I haven't actually looked at them yet, so my comment could be totally false.
4
u/aleixis Mar 08 '17
I remember my Intro teacher saying he was recruited by the military as a teenager. He said Python makes everything else a pinch
3
2
u/dgreenmachine Mar 08 '17
The coding conventions are eerily similar to googles especially in the wording of the last paragraph. You can tell somebody copied it and paraphrased just a little.
8
u/joolzter Mar 08 '17
That's because they are awesome base standards to start from.
1
u/xiongchiamiov Site Reliability Engineer Mar 08 '17
It's almost as if there isn't already a set of standards for python.
2
u/joolzter Mar 08 '17
Of course there are style guidelines. But they aren't a coding standards. Assuming you are referring to PEP8.
1
u/xiongchiamiov Site Reliability Engineer Mar 23 '17
What is the distinction you draw between the two?
2
2
2
u/Guerilla_Imp Mar 08 '17
Most (if not all) antiviruses have a really hard time with dynamic languages, that's why things like Veil Framework exist.
2
1
1
1
Mar 08 '17
If it's used to get rid of Trump excellent, most people on the planet will sleep better tonight. If it's being used to repeat the Bay of Pigs or the assassination of John F. Kennedy no thanks.
1
u/danielsauceda34 Mar 09 '17
What is "python assassin"?
is see pdf build logs for it when reading the PDFs but nothing is coming up on searches
-2
0
-4
-5
0
0
Mar 07 '17
[deleted]
21
u/hovissimo Mar 07 '17
requests is bests
5
3
0
u/jMyles Mar 07 '17
Hmm?
0
Mar 07 '17
[deleted]
7
u/Ph0X Mar 08 '17
I know what requests is, what I don't understand is the other side of your statement, how is requests related?
0
Mar 08 '17
[deleted]
3
u/Ph0X Mar 08 '17
Again, I fully agree with you, but I still don't see the connection to this leak or to CIA.
0
Mar 07 '17
[deleted]
24
u/parkerSquare Mar 08 '17
Plagiarism is a serious accusation.
Much of the text here is from the Google Python Style Guide.
It's only plagiarism if incorrectly claimed as their own. The document clearly attributes some of itself to the Google Python Style Guide, so it's not accurate to call it plagiarism.
10
u/sourcecodesurgeon Mar 08 '17
Also, as someone pointed out above, they're doing development on air gapped systems. If they didn't have an internal version of the style guide, they would need to go to another room just to check the documentation.
And basing a style guide off of Google's guide isn't uncommon. My team's Java guide is basically Google's + internal specific stuff.
3
u/redmercurysalesman Mar 08 '17
Also if you are doing covert operations, you don't want an extremely unique and thus easily identifiable style.
3
0
Mar 08 '17
[deleted]
1
u/trekkiemage Mar 08 '17
They use spaces. Their coding conventions were leaked as well - they use a variant of Google's standards.
0
283
u/Frog_and_Toad Mar 07 '17
I'm ok with it as long as they are following PEP8.