r/Python • u/tc8219 • Apr 25 '21
Tutorial Stop hardcoding and start using config files instead, it takes very little effort with configparser
We all have a tendency to make assumptions and hardcode these assumptions in the code ("it's ok.. I'll get to it later"). What happens later? You move on to the next thing and the hardcode stays there forever. "It's ok, I'll document it.. " - yeah, right!
There's a great package called ConfigParser which you can use which simplifies creating config files (like the windows .ini files) so that it takes as much effort as hardcoding! You can get into the hang of using that instead and it should both help your code more scalable, AND help with making your code a bit more maintainble as well (it'll force you to have better config paramters names)
Here's a post I wrote about how to use configparser:
https://pythonhowtoprogram.com/how-to-use-configparser-for-configuration-files-in-python-3/
If you have other hacks about managing code maintenance, documentation.. please let me know! I'm always trying to learn better ways
37
u/Impronoucabl Apr 25 '21
Beginner/intermidiate coder here, what's a config file?
I think the link got hugged.
My current python project is making a circular gallifreyan translator, but I'm interested if I can use this to my advantage.
20
u/grep_my_username Apr 25 '21
What your program code should contain is how it works.
When there are some data which influence the way it works, then you have some configuration.
Oftentimes, you'll just put them in your code as constants (that word → this word goes in a dict and done). Then you hardcoded your configuration.
But if you put them in an external file, then you get a config file, and it can be changed to make your program do something similar, on other data, with another conf.
Let's consider a gallifreyan translator. Basically, it would take a text as input and ouput another text, only, in gallifreyan (cool project, by the way).
So the process must be something along the lines of classic natural language processing : input, sentence identification, tokenization, lexical transcrition, identification, and reassembly.
Well. At some point here I have a set of rules to match a gallifreyan word to an english one, that are likely in a huge dict somewhere in my code.
If I changed that dict to another one, with all german words, maybe I'll get a rough german translator ? That would be a smart way to reuse my code to make another project real quick !
(Disclaimer: This is not so easy in real life. The hardest part of NLP are grammar, unreliablwe user inptu, and ambiguous meanings. But the idea is still the same)
Incidentally, many many projects use such files for internationalization (i18n): The product I ship at work ships in 18 languages : almost everything the user sees on screen is extracted from a config file of a specific nature : a language file.
Furthermore, you can change user preferences, select options, store tokens of access, etc. Basically every thing the user could change from one run of the software to another should be accessible either in a config file (typically a yaml or ini file) and/or via command-line options.
2
u/Impronoucabl Apr 25 '21
I get what hard-coding is, so is a config file just like a collection bits that could've been hard-coded, but weren't?
E.g keeping to my project, it turns text input into an image output. I'd guess a config file might just contain the different line thicknesses for different 'fonts', vowel spacing, etc?
Am I on the right track?
5
Apr 25 '21
They're still hardcoded, just all in one place. Instead of sprinkled throughout the source code.
→ More replies (1)2
2
u/tc8219 Apr 25 '21
Yes, you could also consider your file output path (if not a parameter), filename pattern, background colors or image files, etc.
→ More replies (2)6
7
26
u/MlecznyHotS Apr 25 '21
Config file stores data needed for the code to run which might vary depending on your use case of the code. Say you have a program that parses a document and counts the number of times the word "like" has appeared. You shouldn't put this word in the code but instead convey it in the config file. It's a better coding practise since your code is universal and work with any word and you specify the word you are looking for through the config instead of editing the source code.
→ More replies (2)1
u/tc8219 Apr 25 '21
Apologies on this.. i was blown away from the discussion. Link should be accessible now
13
u/CotoCoutan Apr 25 '21
Sorry, dumb Q but what exactly am I hardcoding instead of putting in a config file?
6
u/tc8219 Apr 25 '21
totally agree with primary157! Some of my rough guides are:
Anything that may need to change from environment to environment (e.g. between development and production)
Anything that could change if you had to redeploy your code somewhere else - e.g. location of a system path
Anything that you could legitimately update to tweak the behaviour of your application (e.g. number of retrys, or number of seconds to wait before timeout)
Items that maybe an external depedency (e.g. the url for weather API data)
On top of below examples, some others I have in my config file are:
Database paths
Lookup files or initial default values for a database (this might be controversial, but sometimes you need to rebuild your database)
Location of log files
Backup locations
and many more.. it helps to maintain and update your application without having to change your code.
1
35
u/primary157 Apr 25 '21
Actually that's a great question!
But it is hard to provide you a complete answer. I'm giving it a try:
network configuration (ip, port)
external services configuration (ip, port, message config)
parallelism config (number of workers)
enable/disable features
dependency-option (e.g. OpenSSL or LibreSSL)
user-defined config (selected language, remember me, installation folder, region and clock format, preferred resolution, close to tray, enable/disable notification...)
Beyond configuration, the same file formats may be used to store translations (for I-18n support). When handling this manually, they are commonly stored in files separately by language (e.g.
en_US.yaml,ru_Ru.yaml,pt_BR.yaml), each file being a mapping from a default language (key) to the translated one (value).However, there are pretty solid tools that implements I-18n for you (e.g. Qt Framework has a built-in support for multilanguages)
11
u/Pseudoboss11 Apr 25 '21
I've been working on a text bases game. When I started balancing things, I realized how much of a pain it can be to have all the different variables and scaling constants scattered throughout the code.
Even though this isn't supposed to be changed by the user, it is so nice to have it in a config file where I can access them all easily.
3
3
2
5
Apr 25 '21
Basically anything that is specific to a user or to an environment should be in a config file. Assuming I understood your question. :)
1
u/parosyn Apr 25 '21
In python the distinction is a bit more blurry than with C for example (which is a compiled language). For example Sphinx uses python modules for user settings.
6
u/Brothercford Apr 25 '21
Thanks for this! I regularly build reporting that requires config information and I could utilize a single Config for MANY of my projects.
Kudos!
1
4
u/alaudet python hobbyist Apr 25 '21
I have been using configparser for years. From the comments I never realized how strong peoples opinions were on the best way to handle this. I may check a few other options.
1
1
u/alcalde Apr 25 '21
I'm frightened honestly. We're not teaching kids today the basics it seems. I want to sit them all down in front of a copy of Windows 3.1 for a solid week. Have they never heard of System.ini? Once upon a time every setting in Windows was stored in that one file!
That'll knock the Jinja2/yaml/json/environment variables out of them.
4
u/Hi_I_am_karl Apr 25 '21
I agree, but I would argue with reverse issue too. Do not make every hard-coded value configurable by default. Do it only if it gives actual value. It makes the future config file too complex to read.
2
u/tc8219 Apr 25 '21
Totally agree. There are somethings that are still very justified to keep in your code and you just don;t want anyone changing.
One thing I also tend to do is to make sure to add plenty of comments and example values in the config files as well. This is aimed at support teams and also future me that will likely forget what the config was all about!
177
u/troll8020 Apr 25 '21
I use dynaconf. It is flexibility tool for use setting parameters.
73
u/shiba009933 Apr 25 '21
https://github.com/rochacbruno/dynaconf
(For those that want a link)
First time I've heard of this, really neat! Thanks for sharing!
17
45
u/SpaceZZ Apr 25 '21
Is this not just an additional lib I have to import? Config parser is part of std library.
→ More replies (15)28
25
u/abcteryx Apr 25 '21 edited Apr 25 '21
I have been using confuse here and there for simple projects. It was a configuration tool that spun off from a small team making a Python music library manager. Both
confuseanddynaconfallow default key configuration, layers of configs that can override others, etc. But withconfuseI eventually found my use-case to diverge from the one that the music-manager-app devs had in mind.I suspect that
dynaconfis more generalized for use in diverse project architectures.We should also be careful not to optimize prematurely and shoehorn a config library into every project.
A common progression that my code has is:
Script that address a simple, specific problem. I'll show coworkers how cool it is, but they probably won't be able to use it themselves. (<500 lines)
Single-file module that handles that problem a little more generally. Coworkers can use this code if I give them the rundown over coffee. (<1k lines)
pip-installable package with multiple files that fixes my problem more elegantly. I've written documentation and generalized things so people other than coworkers can actually use this. (>1k lines)At Level 1, hardcoded variables (think
WORKING_DIR) are fine.At Level 2, you've identified the things that need configuring, and the things that can remain hardcoded. Consider a config management solution at this point. But seriously, a
config.pythat you import into your main code is probably fine.At Level 3, you actually have a general user in mind. Maybe this is where
dynaconf,python-dotenvand environment vars, or something else comes in. But maybe aconfig.py, supplied by the end-user in their working directory, is fine too!There's a secret Level 4 that almost no project will actually get to. That's the level of legitimate package used by hundreds/thousands of people, actively-developed, etc. Level 4 projects certainly need a more robust solution than a user-supplied
config.py. But if you're at Level 4, then you probably know that already.BONUS READING: The Configuration Complexity Clock. This goes into detail on the different config approaches and their shortcomings. It's a good read for anyone wanting to get a broad overview of the config space.
6
u/mmcnl Apr 25 '21
Why not use environment variables?
13
u/SearchAtlantis Apr 25 '21 edited Apr 25 '21
Because you can stick a config file in git. Environment variables require additional documentation and setup.
As others have pointed out environment variables can be useful for things you explicitly don't want in repositories like keys and passwords.
7
u/reallyserious Apr 25 '21
Env variables are especially useful for sensitive information. You don't want to accidentally push a file with passwords etc to a repo.
→ More replies (4)5
u/tc8219 Apr 25 '21
I'm in two minds. Definitely agree for passwords it's the way to go, but when it comes to moving between environments (development -> testing -> production), then config files are much easier.
19
u/mmcnl Apr 25 '21
I always use
python-dotenvto read .env files. It's very easy and simple. Also suitable for dockerizing.3
1
u/SphericalBull Apr 25 '21
I find env vars a bit hard to set with cron-jobs sometimes
2
u/carloseguevara Apr 25 '21
You can use pipenv in your crontab configuration. And is recommended because not only you will have access to .env file, you will also have the specific libraries (with specific version) for every project in the same enviroment.
→ More replies (1)25
u/vexstream Apr 25 '21
Nice tool but is it really worth another requirement? Maybe I'm a strong outlier here but adding these sorts of simple things as requirements seems.... Offensive somehow. I know requirements don't have a particular cost and it is indeed a tidy interface but it's not that far off from the infamous "is odd" JS package.
9
1
1
u/smokinchimpanaut Apr 26 '21
I agree. For me, every additional component needs to really carry it's weight. If I can achieve something relatively easily with the standard lib, that wins every time. Dependencies do have costs in terms of additional complexity and potential vulnerabilities.
5
→ More replies (1)1
u/RaiseRuntimeError Apr 25 '21
I was going to comment about this, dynaconf is awesome and i wish i would have discovered it sooner. Everett is pretty cool too but not as awesome.
77
Apr 25 '21
I hate ini style configs. Hate. I just use simple variables or dictionaries in a config.py file and import it.
41
u/WillardWhite import this Apr 25 '21
Why not yaml or json?
6
u/zed_three Apr 25 '21
JSON is terrible for human-readable/writable config files. It's much more suited for transferring data between machines/systems/apps whatever.
Lack of comments and trailing commas, mandatory quotes for string keys, way too much punctuation, all make it harder to write than things like yaml (although that also has issues), toml, ini, or other formats
2
u/tc8219 Apr 25 '21
I tend to agree. If it is more for people to manage your applications, who may simply be non-developer support staff, the ini files are easier for them to handle.
19
u/CitrusLizard Apr 25 '21
In my experience, yaml strikes the perfect balance of being both difficult to write for humans, and difficult to read for machines.
→ More replies (1)17
6
Apr 25 '21 edited Apr 25 '21
I do use yaml in a few cases too. JSON less so. Yaml has the one disadvantage of needing a third party module installed but that's usually not much of an issue.
18
Apr 25 '21
Yaml has a lot of disadvantages.
It is far too clever at trying to guess what you mean in a string, so strings like
NO,O13and4:30get unexpectedly translated into a different typeBy default, many implementations silently allow you to store code as well as data. Python is one of those.
A partial Yaml file is still a Yaml file so you have no way to tell if writing is interrupted, or still in process.
Indentation errors are easy to make and hard to debug.
More here: https://noyaml.com/
→ More replies (4)3
Apr 25 '21
That's my only complaint as well. We need a PEP to bring it into the fold :)
→ More replies (1)1
u/twotime Apr 26 '21
For variety of reasons:
Using python allows to refactor common config blocks easily
Python config is directly usable by the caller: you populate the config objects directly and can have accesors/simple calculation, etc, etc
IDEs might be able to help you both when writing AND using config
Last but not the least: python tends to be more readable/unabmiguous than either json or yaml (unless you are sticking to a very small subset of yaml)
21
Apr 25 '21
[removed] — view removed comment
75
u/verdra Apr 25 '21
you don't run any code when you load a json as a dict
importing config.py files can be a security issue.
→ More replies (9)7
u/BosseNova Apr 25 '21
But couldnt malicious code be added to any file imported? Does it really introduce a new risk?
6
u/icegreentea Apr 25 '21
Pretty much. In many circumstances (obviously there are always exceptions), if someone can maliciously modify your config file, they can probably maliciously modify your actual program.
The two better arguments for using serialization languages for configuration is:
Reduced temptation to put logic into your config. Though definitely not bullet proof (looks at yaml...).
Easier for external tools to generate and read your configuration.
→ More replies (3)9
u/PMental Apr 25 '21
Not if you import json files and the like, even if they contained valid python code it wouldn't execute, just be read as data. Importing a script that sets the data up dynamically however means any other code in the file would execute as well.
→ More replies (3)16
u/kinygos Pythoneer Apr 25 '21
More structure to the data, more portable formats, and one thing yaml has over json is you can include comments.
9
Apr 25 '21
In some sense YAML has everything over JSON since JSON is valid YAML. Not a real-world concern though.
→ More replies (2)7
u/Concretesurfer18 Apr 25 '21
Can a config.py update a setting within it that was changed while the program is running like you can with a json?
→ More replies (10)4
u/primary157 Apr 25 '21
Not as easy but it is doable.
Btw this is out of the conversation's scope since they are talking about user defined values is a configuration file.
→ More replies (1)15
u/primary157 Apr 25 '21
Configuration files are mostly better than configuration scripts:
no security issues
language agnostic
if desired it can be updated online at runtime (no need for restart)
cloud-native (depend on the adopted format)
it can be generated by third party softwares (useful for complex config and services integration config)
However for convenience you can implemented a wrapper called
config.pythat load the config file and export python structures of your choice.Note: ini format is platform dependent and isn't standardized. For simple stuff I'd recommend environment variables described in a
.envfile and read bypython-dotenv. For moderately complex config you should use json or yaml. If you're using ini because it supports config sections, then toml is your best choice.2
u/alkasm github.com/alkasm Apr 26 '21
I am not really a proponent of making a config.py file but one huge benefit is the ability to type check your config, which can otherwise lead to runtime failures in your program. A typical way I've seen this dealt with is to have a type-safe default config in your language, and the agnostic config file is used for overriding the defaults.
→ More replies (2)3
Apr 25 '21
Came here to say something similar. I prefer using a
config.pybut there are some projects I work on that need to update their own configs from remote config servers (API or MQ). With JSON orConfigParserthese apps can download the updates and change the config on the fly. Can't do that withpyfiles unless I start pickling. And I avoid pickling like it was Hell itself2
Apr 25 '21
JSON certainly has it's place in the configuration world, it's just not in anything that should be touched by human hands. I lean towards Yaml for most things these days. It'll be a cold day in hell before I'll ever willingly use INI though.
→ More replies (2)4
u/duffer_dev Apr 25 '21
You'll have to then read all the variables individually. A config file gives you a single data structure, most of the time a dictionary, where all the config parameters are stored
7
Apr 25 '21
Putting them into a config.py file and importing it also puts them all into a single dictionary
→ More replies (4)1
2
u/Seankala Apr 25 '21
Is using config files the standard? I just use argparse and put them in the main function.
1
u/tc8219 Apr 25 '21
More of a defacto standard. Many enterprise applications typically use config files as well. I used to be a in team that managed our company's SAP system (a multi-million dollar software license), still uses config files!
Arparse is probably a better option for when parameters are part of direct behaviour of an application or treated as inputs, while config files are more for the inner workings which would not change very often. For example, the input file which contains your data might be a parameter, but then the database server name where you would store the data would be in a config files.
1
1
Apr 25 '21
It's for more complex configs especially where you need to run the app with no CLI args. When you have dozens or a hundred settings, argparse is no real help.
2
u/stefanondisponibile Apr 25 '21
Nice post.
Another common approach I'd suggest is using Pydantic settings management features.
2
2
2
u/kidpixo Apr 25 '21
Nice!l writing!
I spot a typo "it will work in both puthon2 and python 3, which is:"
2
2
u/CleverProgrammer12 Apr 25 '21
I use json files for settings. Don't know if it is considered good practice?
1
u/Isvara Apr 25 '21
That's nothing terribly wrong with it, but personally I find JSON a bit too noisy for a file that's to be edited by humans.
0
u/tc8219 Apr 25 '21
It depends on your usage. I still use json files for initialization values for my databases when they first get created - that makes more sense in a json file rather than config.
Two things to consider about json files is that you cannot add any comments, and that json files might be a bit too tricky for non-developers (e.g. like your suppor teams)
2
u/Fabillotic Apr 25 '21
I‘m hardcoding everything a no one, not a single beast in this inferno of suffering will be able to take back this empire from my constrictive grasp!
2
2
u/Berkyjay Apr 26 '21
Thanks for this thread. It reminded me that I need to do this for a bunch of tools I wrote for my job.
2
u/tc8219 Apr 26 '21
Definitely is a good reminder - I'm blown away about the discussion! Happy to see it helped a few people and also there are some good suggestions of a few other options and libraries too!
81
u/Akmantainman Apr 25 '21
Env variables with Pydantic is also a great out with their BaseSetting class.
16
1
u/andrewthetechie Apr 25 '21
Just started playing with pydantic and discovered BaseSettings. Its aboslutely fantastic.
Pydantic's validation makes it easy to validate your incoming config and with https://sphinx-pydantic.readthedocs.io/en/latest/ you can generate automatic documentation of your config. My current trick is to use @property to add a bunch of config items that are derived from incoming env vars i.e. SQLALCHEMY_URL from POSTGRES_USER, POSTGRES_PASS, etc etc.
Someone suggested Pydantic to me 2 weeks ago and it continues to be delightful to work with. I also have started working with FastAPI and am also really enjoying how well it works.
38
Apr 25 '21
[removed] — view removed comment
7
u/MecRandom Apr 25 '21
Except when you want to share your code, that harder to configure than a (well named) config file
16
u/carloseguevara Apr 25 '21
But, you can share a ".env.dev" file with a template of the data required in you ".env" final file.
9
7
u/Kaligraphic Apr 25 '21
I'd say it's clearer if the template is '.env.example'. '.env.dev' sounds like its the config for the dev environment.
3
u/diamondketo Apr 25 '21
Do becareful and not put absolutely everything in a config file. Things that belong in a config file will organically be there as you develop your project
In the meantime a global variable is a fine alternative
1
u/tc8219 Apr 25 '21
Agree you should be selective in what you put there. If you do put something in a config file, you should expect it to change, for which your application may not be able to handle. If it is something absolutely fundamental to the application or will never change, then you could leave it in your application.
35
u/BrycetheRower Apr 25 '21
python-dotenv is pretty nice too.
→ More replies (2)20
Apr 25 '21
For simple configs this is my choice, too. Json only when things get more complex.
.env files also translate nicely to
docker-compose, which is a nice bonus.1
1
u/BrycetheRower Apr 25 '21
docker-composeis actually what got me into using them! The fact that they're read in automatically makes them super convenient.
2
u/l4adventure Apr 25 '21
Hi what is the benefit of using this vs doing something like "constants.py" and then from your code just doing like
From constants import IP, pwI do this and it works well enough. What is the benefit of using this?
1
u/zenverak Apr 25 '21
I can’t say for sure.. but it’s still better because you never know when your code will grow
1
u/tc8219 Apr 26 '21
Yes, that can work. Some reasons to consider:
If you want to change settings, you will have to change the constants.py which shoudl only be done by developers.. if non-developers are changing the code then it maybe tricky
You do have to 'open up' that constants.py for others to edit - they could add whatever code they like there
With config files, you can keep several versions or several instances depending on enviroment - e.g. config.dev.py ; config.qa.py; config.prod.py
2
u/TheBlackOut2 Apr 25 '21
Pretty easy to just throw configs in a csv and read them from there with good ole pandas
2
u/ericdano Apr 26 '21
Or JSON
1
u/tc8219 Apr 26 '21
definitely an option. Just be mindful of lack of comments and also maintainability by non-technical support staff. If it's just yourself and you're content your futureself will remember, then yes, can work well.
2
u/jim1930 Apr 25 '21
Serious question, is there any reason I would prefer this over a YAML file? In a language like Python for sure, there are tons of options to achieve any kind of goal.
Somebody can say why should I use something else beyond YAML files and just launch the method yaml.safe_load() ?
1
u/tc8219 Apr 25 '21
YAML is also a good option. I have opted for configparser though as the syntax is even simpler for support teams. However, for more intense applications (e.g. ones requiring lists and nesting), yes I've also gone for YAML
1
27
u/licht1nstein Apr 25 '21
Use environment variables and envparse
2
Apr 25 '21
envparse
That hasn't been worked on in almost 6 years, is it fine?
3
u/licht1nstein Apr 25 '21
I don't think it needs any work, it's a simple library that does what it's meant to do. You can write your own envparse in an hour, it's a good exercise.
Anyway, feel free to use any other library.
2
0
0
Apr 25 '21
[deleted]
9
u/licht1nstein Apr 25 '21
Use .env file in development. You get all the advantages of a config file plus the safety and convenience of env vars in production.
Also this: https://12factor.net/
And when testing use test.env
→ More replies (3)1
u/james_pic Apr 25 '21
Part of the issue with environment variables is that they're too easy to just grab from random bits of the code.
You can choose to use them the same way you'd use command line arguments (parse them once at the entry point to the programme, and then pass them into things that use them explicitly - projects like envparse do a good job at pushing you in this direction), but it needs you (and by "you", I mean the whole team) to be disciplined, and not just cave and say "it's going to be a pain to add this new variable to all the things that need it, maybe just this one time I'll grab it straight from
os.environ.It's part of why I tend to lean more towards command line arguments. They're harder to misuse.
4
u/Dogeek Expert - 3.9.1 Apr 25 '21
https://github.com/Dogeek/xdgconfig
Shameless plug here, but I developped that library to handle configuration as painlessly as possible. It's got one object, which uses whatever serializer you want (JSON/JSONC, XML, TOML, configparser or YAML). That objects saves the configuration in the appropriate directory (~/.config/appname or ~/AppData/Roaming/appname depending on the OS) every time it's mutated. It's even got hooks so that you can integrate it in popular CLI libraries (well argparse or typer for now), and it supports versionning the config folder with git (cause I find the feature cool).
1
Apr 26 '21
Why not just put all your variables in a json file and use json parser?
1
u/tc8219 Apr 26 '21
yes, that's also possible. One issue is the lack of comments, and also if your users are not too technical.
1
Apr 25 '21
Hm, I dont know. I tried python-json-config and like the validation features on it. Let's the users define their own variables and validate them easily for their mini apps we hold
1
u/tc8219 Apr 25 '21
python-json-config
thanks - I had not come across this before. The validation feature is very clever indeed.
1
u/Jinkweiq Apr 26 '21
For small or personal projects, just make a settings.py that has a bunch of constants.
1
u/tc8219 Apr 26 '21
for smaller projects, yes, certainly an option. Other option is to have defaults for command line arguments.
Regardless of size, the other key question is who is going to support the app. If it will be others, then the goal maybe to make it as easy as possible for them based on their expertise.
1
u/kielerrr Apr 26 '21
Asking us to learn a new module when we're already to lazy to remove my hardcoded values? gl
1
6
u/i9srpeg Apr 25 '21
I take a different stance. Hard code everything. Then, when you notice something that needs to be different in different environments, put it in a config file. The need must be real and not an hypothetical. We're very bad at predicting the future, so it's better to work on the problems you have today and not on the problems you think you'll have in a year. Putting something that never changes in a config creates accidental complexity, which translates to a harder to understand system, which leads to slower development and bugs.
1
u/alcalde Apr 25 '21
INI files really aren't complex and I can't see them leading to slower development or bugs.
2
2
u/steve986508 Apr 25 '21
This is great, especially all the comments. I'm trying to add a config file for the first time and was having issues even knowing how to begin
2
u/max_tee Apr 25 '21
Just want to mention gconf. I wrote it a to parse and use config files in yml format with minimal effort.
It also supports overrides by environment variables and temporary override of values by a context manager which is nice for unit tests.
2
u/Jmortswimmer6 Apr 25 '21
I would say, know when to use JSON, ini, and .lst files
→ More replies (2)
2
u/MeNotSanta Apr 25 '21
Dropping it here. Use python-dotenv https://pypi.org/project/python-dotenv/
1
u/alcalde Apr 25 '21
How do you save an environment variable? Seriously, you people are scaring me.
1
u/MeNotSanta Apr 25 '21
What do you mean? If you follow the steps in the documentation, the variables you set into .env will be loaded at runtime into the system environments. You can then access them using os.environ[name_of_the_variable]
2
u/cymrow don't thread on me 🐍 Apr 25 '21
Shameless plug for my own: profig. I've tried many config libraries, especially configparser since it's part of the stdlib. But always keep coming back to this one.
2
2
Apr 25 '21
Confog files are good and bad. I use them heavily but they are very easy to overuse. They should be used to make code more simple but can sometimes accidentally obfuscate the code. If you can replace 3+ source files with one source file and 3 configs? That's great. If you turn one source into a source and a config? Not good. Also keep in mind that true jsons have no comments. Also you'll want to come up with a pattern and document your keys and values really well.
Config files are good but easy to get out of hand
1
u/zenverak Apr 25 '21
For sure.
“Let me make everything in the script a variable!”
1
Apr 25 '21
The worst is when people start putting logic in their config files. "All you have to do is run this config file through my python script and it runs the correct commands! Is so modular and dynamic!"
Great so you spent the last 20 hours invent "worse python"
2
u/Surprised_Bird Apr 25 '21
I was trying to solve my problem with ini file and opened reddit just to have some rest. It was the first post... Ok, understood, it's better to go back to my code.
2
u/i_can_haz_data Apr 26 '21
Shameless plug. I created a package that includes a Configuration class that lets you initialize a dictionary-like namespace from a cascade of files (system, user, local) as well as environment variables (including type coercion).
2
1
u/TopoEntrophy Apr 25 '21
Stop using configparser and use .env environment variables instead
2
u/Isvara Apr 25 '21
I like HOCON. A simple, flexible format that lets you use environment variables too.
1
1
u/dustractor Apr 25 '21
or make your program take command line arguments with argparse and use your ide or makefiles or whatever idk
→ More replies (3)
1
u/metaperl Apr 25 '21
ConfigObject greatly improves on configparser. IIRC, ConfigObject still falls short of Dynaconf.
But for me, I prefer Pydantic settings over anything else by a long shot.
1
u/Equivalent-Wafer-222 Technical Architect Apr 25 '21
I like this one, it auto-loads all active files in my settings folder which I can import from from any other location of the app. Most of my config files are json but I can see this easily adapted for yaml as well.
1
u/AnomalyNexus Apr 25 '21
Is there something vaguely comparable in the standard library?
3
1
u/the_real_jb Apr 25 '21
I really like OmegaConf, which has great utilities for merging multiple configurations. Seems to mostly be used in machine learning
1
u/awesomeprogramer Apr 25 '21
You should take a look at modularyze, which aims to allow dynamic and modular configs by using jinja2 as a templating engine on yaml files.
1
u/alcalde Apr 25 '21
WHY? That sounds... incredibly over-engineered.
1
u/awesomeprogramer Apr 26 '21
Because we can! /s
Because it's yaml based, it can instantiate basically anything, so really, parseing a config is akin to loading a module. It's really quite nice to use actually. The source is also surprisingly short, so not too over engineered.
1
u/pokk3n Apr 25 '21
I switched to Pydantic for config files since I like it for models. The way I've got it set up you just set your env variable in your environment and it picks up whatever 'env.env' (e.g. 'prod.env') file you have set and populates the config object based on that. Not quite the ideal world of the whole modern environment variable driven stuff but we aren't containerized so have lots of scripts running in one environment, so files work a bit better for us (vs. having a gajillion environment variables with prefixes to separate them).
1
u/ijpck Apr 25 '21
How do I use config files if I can’t have them with all that sensitive info in GitHub?
1
Apr 25 '21
It's true the stdlib
configparserdoesn't handle secrets (or types). I've made extensive use of it, and just wrapped it in a custom class that casts to the desired type, or encrypts/decrypts based on a secret key not included in the config file.I haven't tried
dynaconfbut it looks awesome, and it does do all of that. Including secrets1
Apr 25 '21
Keep them in a separate, private repository. Nothing wrong with having sensitive info on Github as long as access is controlled.
1
u/Supadoplex Apr 25 '21 edited Apr 26 '21
and the hardcode stays there forever.
Or you know, until there is a need for it to change.
From the linked article:
One reason that .py files are not used is that Python 3 can be slower when it comes to reloading. You would need to restart the whole program if you stored your config in a python .py file.
Doesn't importlib.reload work for this use case?
Also, the end user can modify the code at will if it is in .py format.
Isn't the ability to modify the code a benefit?
End user can modify rest of the code as well, and whether you use Python or another language for config has no effect on that.
1
u/grady_vuckovic Apr 25 '21
Personally I do all my config files in json these days. In either Python or Node.js, it usually only takes half a minute to make a function that reads a .json file during startup, and parse it into an object/dictionary. Easy.
1
Apr 25 '21
Can somebody explain to me what .ini files have to do with hardcoding? All they do is store variables, right?
1
u/Isvara Apr 25 '21
They store values, yes. Instead of hardcoding them.
1
Apr 25 '21
Oh I see, but what's the point?
3
u/Isvara Apr 25 '21
Firstly, it means that someone who wants to configure how your code works doesn't have to look through and modify your code. The parameters of your program are in one place.
Secondly, it means that the same code can be run with different behaviors, by invoking it with different configuration files.
1
1
u/fried_green_baloney Apr 26 '21
Particularly with source control. Where you have one of the Python files with dozens of changes mostly between num_trials = 10000 and num_trials = 50000 and back to num_trials = 10000 again.
Also command line arguments. Even environment variables better than editing source all the time.
PS: This has elements of "do as I say, not as I do", but I've experienced the pain.
1
u/fukitol- Apr 26 '21
I don't really even bother with configparse. A yml or json file suits 99% of my use cases. I'll give it another look.
1
1
u/cburnett_ Apr 26 '21
There are a couple environments where ENV variables make a lot more sense. Otherwise I just fall back to yaml or json.
1
u/adrielquinn Apr 26 '21
I have a standard practice writing all my config files in the ammo for it’s human readability yank. It was super easy to write and import a tiny class to read yaml config files using ruamel.yaml across all my projects. So that’s another option
1
u/neondirt Apr 26 '21
But let's be honest, the work involved by making things configurable isn't dominated by parsing a file.
1
1
Apr 25 '21 edited Feb 05 '24
Reddit Moderation makes the platform worthless. Too many rules and too many arbitrary rulings. It's not worth the trouble to post. Not worth the frustration to lurk. Goodbye.
This post was mass deleted and anonymized with Redact
→ More replies (2)1
u/alcalde Apr 25 '21
In fact, it’s best to write it yourself to know well how it works.
That doesn't sound very pythonic and... who cares how a parser works? It parses. You're just trying to save settings.
2
Apr 25 '21 edited Feb 05 '24
Reddit Moderation makes the platform worthless. Too many rules and too many arbitrary rulings. It's not worth the trouble to post. Not worth the frustration to lurk. Goodbye.
This post was mass deleted and anonymized with Redact
0
Apr 25 '21
Use environment variables!
5
u/CitrusLizard Apr 25 '21
But which environment variables? If only we had some sort of file that could tell us!
→ More replies (1)→ More replies (2)2
Apr 25 '21
There are valid reasons to.choose JSON, yaml, or INI versus environment variables or py files is that in the latter cases, the settings are read-only. They are, effectively, hard-coded just not in the business logic.
Perfect for 95% of use cases.
But there are cases where the app, or a related app, needs to change the config itself. It needs to write to the config file itself, not just change the value in memory.
I have one project -- a remote agent -- that uses env vars, INI, and JSON. Env for preconfig for docker, or enabling debug mode on the fly. JSON for several remotely configurable processes. INI for the app config itself, which can be remotely overridden.
0
u/aka-rider Apr 25 '21 edited Apr 25 '21
Config files in programming languages without compilation artifacts have zero fucking sense.
Just use separate file as config, and that’s it. It gives you possibility to be as explicit as you want while not repeating yourself.
You can do
PORT = 12345
URL = “hostname:” + str(PORT)... and so on.
Instead I see another shitty yaml with copy-paste all across.
Why on Earth do you want to keep separate config, then repeat same structure in the code, then have separate serialization/deserialization step plus validations?
This is repetition, repetition, repetition, and a huge source of bugs.
2
Apr 25 '21
Until your app needs to change those settings and write it back to file.
Swear to god all the "use this read-only approach" posts are by web devs or wannabes. The ability to automate configuration is kind of a big deal for some applications
4
u/alcalde Apr 25 '21
Swear to god all the "use this read-only approach" posts are by web devs or wannabes.
Thanks; I've been writing code to one degree or another for 30 years - no web stuff - and I've been reading many of these comments with wide-eyed horror.
"Save everything in environment variables!" Great; and what if another program has the same idea and uses the same environment variable name? Or a user decides to reset or change shells and deletes their shell's config file?
It's madness. Pure madness.

74
u/duffer_dev Apr 25 '21
Over the years I have tried these config files.
JSON : simple, readable and easily translates to dictionaries
YAML : mostly like JSON but has additional features like comments.
TOML - unlike YAML and JSON, indentation is not a pain. However, complex structures like lists of lists can be slightly tricky
INI - much simpler than the above three. Lists/arrays can be tricky, but still can be done.
All the different configs translate to dicts in python. The kind of config also depends on your task. Something like some configs with few parameters, I'd suggest INI as that is much simpler. But for something more complex, like a ML pipeline or data science project, YAML would be more suitable.