r/programming • u/[deleted] • Jan 28 '19
C++ Modules Might Be Dead-on-Arrival
https://vector-of-bool.github.io/2019/01/27/modules-doa.html143
u/WalterBright Jan 28 '19
D made the decision at the beginning to have filename==modulename specifically to avoid this problem. So:
import std.stdio;means look for:
std/stdio.d97
u/vector-of-bool Jan 28 '19
This would be my preferred solution. Simple, hierarchical, easy to understand, and it works for many languages.
147
Jan 28 '19
Which are all the reasons it won't be like that in C++
35
14
u/Valmar33 Jan 28 '19
Why must they inflict their misery upon others? To pretend at being clever and intelligent? To remain relevant? To inflict mental torture on people trying to figure out to how solve programming conundrums?
I guess they really want to help C++ programmers keep their jobs, because otherwise, their huge projects would implode.
Kind of like Java ~ except replace long-ass class names with tormentingly complicated grammar that explodes horrifically if you miss a step.
Those
ranges-v3templates come to mind... :|15
u/Patman128 Jan 28 '19
Why must they inflict their misery upon others?
If a good idea breaks the workflow of even one of the Elder Wizards on the C++ language committee then it's thrown out.
12
u/lord_braleigh Jan 29 '19
Note that the "Elder Wizards" often work on the behalf of major tech companies, and influence the C++ standard so that these companies won't have to break their workflow or make sweeping changes.
This isn't necessarily a bad thing - backwards compatibility is one of C++'s selling points, and there are plenty of shiny new languages you can use instead if you don't care about that.
14
u/SAHChandler Jan 28 '19
A shame that some operating systems like zOS don't really support large hierarchies. Hence the existence of P1302 which supports a "splayed" approach :v
44
u/jl2352 Jan 28 '19
Having zOS cripple modules for everyone else is pretty ridiculous.
In the web world we had this with IE. So we said fuck IE, and everyone else moved on.
11
u/SAHChandler Jan 28 '19
While that might work in other places, the vendor representing zOS is also the representative for a national body (IBM Canada). While many people would love to see zOS go (the last of the EBCDIC systems), it's not going to happen for a while, especially when at this point there are many systems out there (e.g., cars on the road in America. I should note: zOS is not crippling modules. It just removes the ability to state we can have directories represent module hierarchies. But we can still have directories represent a module, which is how most languages work nowadays. Build systems can then enforce this naming scheme. It's a bit more work for build systems authors, but I'll take that over "write an entire dependency scanner that needs to know about N interface modules and M implementations modules in every project with no indication of what is located where" which is what the modules TS gave us.
5
u/Holy_City Jan 28 '19
Aren't those usually cross compilation targets anyway?
1
u/SAHChandler Jan 28 '19
From what I understand development happens directly on some zOS mainframes :(
-7
u/diggr-roguelike2 Jan 28 '19
So we said fuck IE
No you didn't. Microsoft did, and you just got lucky off their incompetence.
Google won't be so braindead this time around the carousel.
0
Jan 28 '19
[deleted]
3
u/curien Jan 28 '19
I think they're referring to MS basically feature-freazing IE for over five years after the release of IE6.
8
u/Valmar33 Jan 28 '19
You could still do something like D is doing, no?
Just modify things just enough to support those environments.
Maybe
std_stdio.cppor something. Add more underscores to emulate deeper hierarchies.That is, if I'm understanding your statement correctly.
7
u/SAHChandler Jan 28 '19
Yes we can. I wrote P1302 to solve this problem. While it's not intended to add a 100% hierarchical module system, build systems are free to enforce this, while letting those on other systems do as they please while also allowing large existing code bases to slowly move to it.
12
u/kuikuilla Jan 28 '19
Would that forbid you from having a
std.dfile at the root? Where would you put stuff that belongs in the std module? Or does it have the possibility of having anstd.dfile and anstdfolder?16
u/WalterBright Jan 28 '19
This is done via the package system.
7
u/kuikuilla Jan 28 '19
So you'd have a file at
std/package.dwhich would contain common stuff belonging tostdmodule? Rust used a similar system where you would have amod.rsfile, but it was kinda annoying because then in a project you would have tons of differentmod.rsfiles. It made navigating in IDEs hard, you had to check the full path of the file you were trying to edit/open/view in order to know what it actually was.6
u/WalterBright Jan 28 '19
In D this system works well. It resolves the ambiguity between a folder and a file, and we came up with it as a backwards compatible improvement to enable large imports to be broken up into multiple smaller ones.
7
u/kuikuilla Jan 28 '19
So it's not cumbersome to have multiple package.d files?
For comparison, Rust moved to a system where you can have
foo.rsat root (or any level really) and then have a folderfoo/for the contents of that module.├── foo.rs ├── foo │ ├── bar.rs │ ├── foobar.rs │ └── fizzbuzz.rsThat way it's imo easier to navigate to the code of the foo module because you don't need to type in the path of the module file when you go to a file in some IDE.
5
u/WalterBright Jan 28 '19
multiple package.d files
Not in the same directory.
you can have foo.rs at root (or any level really) and then have a folder foo/ for the contents of that module
Can't have both in D.
foo/package.dtakes the place offoo.d.I'm not aware of any prior art with the way D does this. Users are happy with it, though, it seems very satisfactory.
5
u/kuikuilla Jan 28 '19
Not in the same directory.
Yeah I was just wondering if "go to file" and such functionality in IDEs would be hampered by files with identical names. I didn't like it when working on a project using Rust.
4
u/renstarx Jan 28 '19
I don't know when D started using this structure. Python does it the same way.
foo.pyandfoo/__init__.pyboth createfoomodules. Now I am curious which one came first, and why this is so complicated for C++ to follow...I would hazard that you cannot have both in the same path, but that I don't know.
1
u/steveklabnik1 Jan 28 '19
then have a folder foo/ for the contents of that module.
... then have a folder
foo/for the contents of any sub-module offoo's. The contents of thefoomodule itself are still in afoo.rs.2
u/kuikuilla Jan 28 '19
Yes, I know. The new way of handling modules and the paths is way better in 2018 edition.
3
u/adr86 Jan 29 '19
except D doesn't actually do that. you should know this!
the module declaration is what gives the name. the filename is just the search path, but you can pass any file to the compiler.
1
u/WalterBright Jan 30 '19
It's true if you pass the filename one the command line, it can differ from the module name. But the vast bulk of files are not given on the command line, and the import is looked up by using the import name as the filename.
2
u/flukus Jan 29 '19
How does this work in atypical scenarios, like when you're generating source code to a temp directory?
3
u/WalterBright Jan 30 '19
You can use the command line switches to set the path search, and even remap the module names to actual file names.
-17
u/idle_zealot Jan 28 '19
Did you leave this exact comment in the HN thread too?
49
-22
Jan 28 '19
Nobody cares about your language, Walter.
6
u/Valmar33 Jan 28 '19
It's better than C, that's for sure. C++ simply has a bigger ecosystem, and that Network Effect is very hard to overcome.
Bounds-checking is a huge win. So is being able to convert raw pointers to bounded arrays with slices.
The language has wrinkles here and there, but the benefits make it worth it.
I may have chosen Rust, but the syntax was too foreign compared to D for me to be comfortable with, and I like playing with memory as I try to understand game engine programming. So, D it was. Go... yeah, I think most readers here would be aware of why Go is not a replacement for C nor C++.
27
Jan 28 '19 edited Jan 28 '19
Provide a “manifest” or “mapping” file that describes the filepath to an MIU based on the module name. This file will need to be user-provided, or we are back in the scanning problem.
Why can't this be <implementation-defined> just like the mapping from header file inicludes to files?
Honestly, what you propose won't fly because (1) you are turning C++ compilers into C++ compilers + build systems, and (2) because getting 150 countries and >100 committee members to agree on formats for these manifest files, file system structure, etc. is pretty much impossible.
Also, many languages have solved this problem, and none of them had to turn their compilers into build systems.
One such language is Rust. C++ modules solve the same problem as Rust crates. The Rust language provides an import syntax for crates (extern crate foo), and the user provides the mapping from that to files manually by passing CLI arguments to the compiler - the dependency must be compiled before, so this happens-before relation exists there too.
Nobody provides these CLI arguments manually. Instead, the build tool (cargo) parses a TOML manifest file that specifies dependencies, builds a dependency graph, fetches them from the internet, the file system, etc., compiles them, and calls the compiler with the right arguments.
Another build tool could parse a YAML manifest, or parse all source files scanning for imports, and generate the manifest file automatically (this would be incredibly hard to get right for Rust, and probably impossibly hard for C++ though), but still reuse the same compiler by calling it with the right arguments.
Most importantly, neither the compiler nor the language have to know anything about this.
In fact, as you have noticed, happens-before requires you to compile dependencies before hand. That is, one needs to pass the modules that each module imports to the compiler "somehow". That's serving the same function as import module. So in Rust2018, you don't even need to import crates. Crates passed to the compiler via CLI are put in the root namespace, and that's it, you can just use them without extern crate. If you forget to pass one to the compiler, you just get an error because the path cannot be resolved.
20
u/pjmlp Jan 28 '19
Apparently because many vocal devs don't want to change their ways to adopt modules.
At least is what it feels to me standing on the sidelines.
14
Jan 28 '19
They are already passing this information to the compiler via include paths, library paths, etc. "somehow" (e.g. CMake might be doing all the heavy lifting). They might be able to adapt their build systems to pass module information as well.
If what they want is for the build system to automatically generates the manifest files for them, they are probably out of luck. Doing so is pretty much equivalent to extracting which dependencies C++ code needs from the C++ code itself. You would need a C++ compiler that understands C++ and macro expansion for that, but doesn't need dependency information because it is not available yet.
This would be a very hard problem to solve for any programming language, and probably an impossibly hard to solve for C++.
2
7
u/SAHChandler Jan 28 '19
Just a few small notes here. Not trying to nerd snipe you, just want to add some info.
Rust’s compiler is in fact a build system, as the entire crate is treated as a translation unit. This is why incremental builds were an issue for a while. I spoke with several core people at RustConf 2018 when I was trying to see what solutions could be made if we can’t make the C++ compiler a build system, as compiler vendors are very against the idea.
while ISO is a very large organization not all members participate in WG21. We have about 15 national bodies (possibly a few more I need to go back and count the exact number), each of which has one vote. Each national body is made of various members from said country, though some permit anyone to join as long as they have the cash (such as the United States via INCITS) each committee member therefore has a fractional vote unless they are the only member of their national body. Thus far, there aren’t many that do this.
C++ is older than I am and there is code out there still compiling that was probably written before I was born (and I’m going to be thirty next month if I can out my age and also give additional context). The oldest code base I’ve worked on was written when I was about 6 years old. While everyone can be up in arms and shout “maybe it’s time to break that old code”, many on the committee don’t wanna. I’m in favor of it, but you’d have to convince several national bodies to go forward with it.
0
Jan 28 '19 edited Jan 28 '19
Thank you for the comments, not taken as a nerd snipe at all.
Rust’s compiler is in fact a build system, as the entire crate is treated as a translation unit.
Rust crates are single compilation units, very similar to C++ translation units (TUs). Rust modules are more like C++ namespaces.
When one talks about incremental (or parallel) compilation in Rust, what they are talking is about compiling different parts of a single TU in parallel. This is something that AFAIK no C++ compiler does. If you want to compile C++ in parallel, you need to split the code into independent TUs. In the same way, if you want to compile a lot of Rust code in parallel, you need to split it into multiple crates that can be compiled in parallel - the parallelism that one can extract inside a single TU is not enough (and incurs other obvious cost like having to put the whole content of the whole TU in memory).
Comparing C++ modules to Rust crates isn't 100% apples-to-apples because C++ modules can be split into many TUs. Still, C++ modules consist of a single TU in the worst case (which is not uncommon nor a bad thing, and often, Rust crates can also be split into smaller crates.
So the claim that the Rust compiler is a build system isn't really correct. Yes, if you split a single TU into different files, Rust offers you a way to include all those files together into the TU that gets compiled as a unity, and the Rust compiler handles that. But the Rust compiler does not handle finding/fetching/resolving/compiling other TUs, that's all handled by a build tool.
2
u/flatfinger Jan 28 '19
Honestly, what you propose won't fly because (1) you are turning C++ compilers into C++ compilers + build systems, and (2) because getting 150 countries and >100 committee members to agree on formats for these manifest files, file system structure, etc. is pretty much impossible.
A good standard for a language should seek to maximize the number of tasks which can be encoded entirely using set of text files whose meaning is specified by the Standard. If someone who is familiar with the implementation is given the set of files, they should not need to know anything beyond the names of the files in order to make the implementation either build an application, refuse to do so if the files require features the implementation doesn't support, or spend a possibly-unbounded length of time deciding what to do.
Efficient development rebuilds may require a more sophisticated build system whose details could and probably should vary among implementations, but a program that's written in C but which would require that anyone running it on an implementation other than those the developer knows about mess around with with build files and other such things in order to use it at all isn't really a "C program"--it's a "C and whatever build systems the programmer happened to support" program.
1
Jan 28 '19 edited Jan 28 '19
A good standard for a language should seek to maximize the number of tasks which can be encoded entirely using set of text files whose meaning is specified by the Standard.
I disagree with the premise. This is just one of the trade-offs involved, and I think this trade-off is important enough to be factored and balanced with other trade-offs. But there are many trade-offs, and I don't think this one is the most important.
Particularly, because it assumes that the standard is a C++ manual, which arguably is something I can understand, since many people nitpick on what "correct" C++ is based on the standard. But the C++ standard is not a C++ manual for users, it is a C++ manual for implementers.
No user should need to know what exactly the standard says about anything. The only thing most users need to know is how to use C++ to solve a particular problem in the platforms they care about, with the compilers, build systems, etc. they care about.
And as long as the build system behaves in some sane way in the platforms an user cares about, its pretty much irrelevant whether the standard talks about this or not. The C++ standard says nothing about CMake, Makefiles, etc. and people are still able to use C++ to solve useful problems.
I mean, the standard pretty much say nothing about include files. An implementation could require users to screenshot source code and save it as png images for their compiler to include it properly, but in practice, most implementations don't do this. Instead, they provide sane ways based on the file-system, environment variables, etc. of controlling what is included and how.
1
u/flatfinger Jan 28 '19
The only aspect of the build system I'm asking the Standard to care about is that it have some standard concept of "process this bunch of source files in the Standard fashion", and that it be possible to accomplish most tasks using programs for which that build method would suffice.
If I want to write a program that doesn't make use of any special features of e.g. the Macintosh platform, the Windows platform, nor the Unix platform, it should be possible to express that program in such a way that someone on any of those platforms could easily build and run it without the programmer having to know anything about the target platform, and without the user having to know anything specific about the program beyond the names of the files involved. If there's no interest in running a program on implementations the programmer doesn't know about, why bother with a standard? What purpose would it serve?
1
Jan 28 '19 edited Jan 28 '19
"process this bunch of source files in the Standard fashion"
The standard doesn't really have a concept of a file system, files, directories, nor of the many other things that could play a role here (directory structures, symlinks, etc.).
Somebody would need to add all of that to the standard, making peace with pretty much every platform that pays someone to assist to committee meetings ("But our filesystem has no directories", "But our file-system has no files", "But traversing directories looking for files is too expensive in our file-system", etc. )
why bother with a standard? What purpose would it serve?
Every single compiler and platform out there has platform specific extensions that are not in the standard. The standard is just the bare minimum that must be supported by all platforms, and keeps C++ from diverging into completely incompatible dialects. Writing code for different platforms always has, and for the foreseeable future will require, writing platform specific code.
Because C++ is used to target widely different platforms, the standard has avoided specifying pretty much anything about how the C++ source is input into the compiler. There is no standard C++ file extension, some people use
.cpp, others use.cxx, others use.cc, etc. The standard does not specify this because not all platforms have files with extensions and there is people using C++ to target those (if you look carefully you'll see that standard library includes have no extension). When you put a path in an#include path/to/header.hthe standard doesn't say anything about what goes where the path belong, only that there must be some implementation-defined mapping between what you put there, and some other source code, and the semantics of the include (including the contents of whatever is in the path verbatim). Yet in most platforms, includes work like all users expect, e.g., following file-system paths relatively.Basically, just because something isn't in the standard, doesn't imply that 99% of users can't rely on it being true 100% of the time. However, if you fix the standard to something that 1% of users can't use, then with a million C++ developers you'll get 10.000 people voting against the proposal and sunking it down. The standard doesn't evolve with things that have 51% of the votes, only stuff that has 100% of the votes goes in. This is why C++ has so many users.
1
u/flatfinger Jan 28 '19
Of course one will need to use platform-specific code when targeting features of specific platforms. One should not, however, need to write a bunch stuff that's specific to particular implementations when one isn't using any special features thereof.
If some people would need a compiler to process things in a fashion different from what other people would require, the solution should not be to simply have the Standard say that any implementation may treat the construct in any fashion whatsoever, but instead say that:
If a program contains a directive explicitly calling for some construct to be processed in the common way, a compiler shall either behave in common fashion or reject the program.
If a program contains a directive explicitly calling for some construct to be processed in an uncommon way, a compiler shall either behave that requested fashion or reject the program.
If a program contains neither kind of directive, an implementation may assume as a default whichever behavior its author believes would best serve its users.
The fact that 99% of programs opt to use the common behavior should not prevent the remaining 1% from employing some alternative on any compilers that opt to support it; nor should it prevent all 100% of the programs from having a defined meaning that would not depend upon the target implementation (implementations may be unable to process a program that demands unsupported features, but behavior would be limited to rejecting the program as opposed to formatting hard drives, releasing nasal demons, etc.)
Basically, just because something isn't in the standard, doesn't imply that 99% of users can't rely on it being true 100% of the time.
It doesn't imply that 99% of users shouldn't be entitled to expect that it would be true 100% of the time on commonplace non-obtuse implementations. Unfortunately, the authors of the Standard regarded the notion that quality implementations should Serve the needs of their customers when practical, whether or not the Standard requires it, as being self-evident and failed to consider that obtuse implementation might use the Standard as an excuse to do otherwise.
1
Jan 29 '19 edited Jan 29 '19
It doesn't imply that 99% of users shouldn't be entitled to expect that it would be true 100% of the time on commonplace non-obtuse implementations.
That's pretty much what "implementation-defined" means. As I said, the standard isn't a C++ manual for users. Nobody uses
#include file/pathand thinks, "oh damn, I'm relying on implementation-defined behavior, my code isn't portable".1
u/flatfinger Jan 29 '19
According to the published Rationale document for the C Standard, it's also what Undefined Behavior was supposed to mean in many cases, since the authors regarded the question of how to process it (e.g. whether to process it in a documented fashion characteristic of the environment) as a Quality of Implementation issue. The authors of the Standard intended that implementations would support "popular extensions", and such a term can't sensibly be applied to Unspecified Behavior or Implementation-Defined behaviors. The fact that an implementation documents something that it's required to document, or that it happens to document the fact that it chooses from among specific actions described by the Standard doesn't seem like much of an extension, compared with providing means of doing which the Standard does not require them to define.
A fix would be to define standard macros that would allow a programmer to start a program with a directive like:
__STDC_LIMIT_UNUSUAL_INT_MATH_EFFECTS(EXTENDED_TEMP_ONLY)and then know that any implementation would be required to either 1. guarantee that integer math other than divide/remainder will never have any side-effects beyond yielding some kind of integer value, but temporary values of signed integer types need not be truncated to their actual size, or 2. reject the program entirely. The question of whether an implementation should support the requested semantics would be a Quality of Implementation issues, but the question of whether an implementation could jump the rails wouldn't be.
0
Jan 28 '19
[deleted]
11
Jan 28 '19 edited Jan 28 '19
It's hard to tell what point you are trying to make. But if you are trying to state that Rust approach is hard for regular programmers, I strongly disagree.
Pretty much no C++ programmer actually knows exactly how dependencies are added to their project. The reason for this is that most of it happens implicitly. Yes, Makefiles, CMake files, etc. express some of the dependency information, but a lot of the information is passed implicitly to the compiler via environment variables (e.g. for the system header files, system library files, etc.), some of them also rely on conventions (e.g.
#include <>vs#include "", etc.), the order of paths in these variables is relevant, etc.Things magically appear to work for C++ until they don't, e.g., wrong library version / path being "magically" picked, wrong standard library picked (e.g. using clang+libc++ on Linux for one project when most of your other projects use gcc+libstdc++), etc. Debugging these problems typically require getting the compiler into a verbose mode for some TU which dumps all include paths, library paths, etc. being considered, their order, which one got picked and why, etc. Most devs just fickle with their environment till things work, they are probably even unaware that tools to debug these issues exist. I've seen C++ devs waste days trying to fix linking / include issues, and weeks trying to fix build system issues.
There are some Rust features that are too implicit for newbies to understand, like Rust2015 crate paths, but dependency resolution / the build system is not really one of them. Rust projects have an explicit
Cargo.tomllisting their dependencies and that's pretty much it. Is the wrong version of a library being picked? Open theCargo.toml, change the library version/path/git-branch/..., done. Newbies don't, in general, know anything about the dependency-resolution algorithm used, but because it is all explicitly listed in one place, they can manually change things if the algorithm produces an output that they don't like.8
u/pakoito Jan 28 '19 edited Jan 28 '19
Sarcasm doesn't transmit well on internet posts, that's why I tried to put it as a strawman quote.
3
2
17
Jan 28 '19
[removed] — view removed comment
0
u/flatfinger Jan 28 '19
The concept of datestamp-based make files is a useful hack for incremental builds of some kinds of languages, but it can't efficiently handle situations where the inputs to a build step are likely to change in ways that don't affect the output. If one were to make each build artifact include date stamps and hashes of all the input files and end with a hash of any content that would affect files depending upon it (including the hashes, but not timestamps, of files it depends upon), that would allow build systems to be much more efficient [if the date stamp of a source file precisely matches the cached value, there's no need to look at it. Otherwise, read or compute the hash of each input file and see if it's changed. If any have changed, rebuild the output file. Otherwise simply update any cached timestamps.
1
u/oridb Jan 29 '19
Hashes help less than you'd think. Updating a file in almost any way changes the output hash. Even changing blank lines changes the offsets into the file used by debug info. And even in prod builds, there's debug info for unwinding the stack when propagating exceptions, printing stack traces, and so on.
1
u/flatfinger Jan 29 '19
It's common for header files to be included by dozens or hundreds of other files, most of which only need to use a few declarations. If one were to, start compilation with a parsing phase followed by a stage that omits any unused declarations, changes to header files wouldn't affect the output of the parsing phase in most cases other than those where the changes would affect the semantics of the program. Handling debug line numbers may be a bit tricky, but that could be handled by having debug line number in a separately-hashed piece of the file, and having each stage produce a back-mapping file. Not a trivial design issue, but one which should have been solvable ages ago.
1
u/oridb Jan 29 '19 edited Jan 29 '19
It's common for header files to be included by dozens or hundreds of other files, most of which only need to use a few declarations.
Correct. However, the headers produce entries in the debug info. So, you either get incorrect line numbers and unwind info for macros, template expansions and similar, or you don't actually win anything.
Handling debug line numbers may be a bit tricky, but that could be handled by having debug line number in a separately-hashed piece of the file
And now your build system isn't just language dependent -- it's compiler dependent! And, very likely, compiler version dependent. And god help you if you want to cross compile and link with other languages through an FFI.
1
u/flatfinger Jan 29 '19
Line numbers measured from the start of a file are not a very good way of identifying locations within one. If a program with debug info says an error occurred on line 493 of woozle.c, that may not be terribly helpful if anything might have changed above that line since the time that version of the code was built. If, however, a location was recorded as "the 493rd token of template-expanded code encountered during the compilation of foo.c, whose template-expanded code would hash to [xxx]", such a location could be relatively quickly mapped to a location in the source file even more precisely than with normal debug info today, even if there were changes in source or header files above the portions that were used in the function.
The first Pascal compiler I enjoyed using was Turbo Pascal 2.0. If a run-time error occurred, one would get a message like "Run-time error 23 at address 49A3. That wouldn't seem like a very useful message, but if one fired up the IDE, selected "find run-time error", and typed in 49A3 one would get taken to the location in the source code where the error occurred. This wouldn't likely work if one had made changes to the semantics of the program, but would be unaffected by newlines, which seem to be the nemesis of the file-based line-numbering approach.
Thus, what I would envision would be a system where debug information is recorded in function-relative fashion, and then if desired it is sorted and the early stage of the compiler is run again while observing the physical line numbers where each function-relative position is encountered. Changing the text of a function or any macro template that is actually included in the function would require recompiling that function, but changes to macros and templates that are not used within a function would not require any recompilation thereof.
To be sure, no incremental build system is going to be perfect, but a system based upon hashing the semantically-relevant parts of each program could nonetheless avoid a significant fraction of the build times associated with date-based make systems.
BTW, even if one didn't want to change tooling to use function-relative token counting, I think one could reduce the extent to which changing files changes reported line numbers if one added artificial
#line 10000,#line 20000etc. directives within source and header files, and recognizes that a reported line number 20123 means "the 123rd line following the#line 20000directive". The addition or removal of lines between the#line 10000and#line 20000directives would only affect line-number debug info for files that actually used some portion of the header that had appeared between those two directives.1
u/oridb Jan 29 '19
Good luck on convincing all compilers to make abi breaking changes and moving to a new debug format that doesn't quite solve the problems you think it does in order to save a few minutes of build time on purely whitespace changes.
It's only a few million lines of churn across a few hundred projects. No biggie.
1
u/flatfinger Jan 29 '19
I've never had to deal with large C++ projects, so I don't have personal experience with their build times, but I've heard people who sing the praises of C++ talk about 15-plus-minute build times. Meanwhile, many other languages seem to be designed to minimize build times, and I think that is likely a significant reason for their success.
Why was Pascal a dominant programming language on the PC? It became dominant because on a system without a hard drive, the edit-run cycle on Borland's Turbo Pascal IDE was often an order of magnitude faster than any non-interpreted competitor. Javascript is a really crummy language, but node.js can manage to process it efficiently enough that the value of further performance improvements would be limited. From what I understand, go offers even better execution performance while still being fast to build.
It used to be that the only way to get adequate performance from an object-oriented language was to use C++ and put up with slow build times. As the performance of competing languages has improved, however, I see less and less reason for programmers to favor a language with slow build times over a faster one. Maybe the notion of improving C++ build times is moot because projects that can use languages with faster build times have already been migrated to those languages and wouldn't go back to C++ even if its build times were made competitive.
1
u/oridb Jan 29 '19 edited Jan 30 '19
I've never had to deal with large C++ projects, so I don't have personal experience with their build times, but I've heard people who sing the praises of C++ talk about 15-plus-minute build times. Meanwhile, many other languages seem to be designed to minimize build times, and I think that is likely a significant reason for their success.
15 minutes isn't too bad. Tensorflow takes closer to 2 hours for a clean build. Same with LLVM. LLVM is part of the OpenBSD base system, and takes longer than all of the C code in the rest of the system, at least on my machine.
The problem is that individual files can sometimes take multiple minutes to compile, and huge amounts of semantically significant code actually lives in headers, which means you can't get away /without/ recompiling some dependencies. "false recompiles" aren't a huge problem in the projects I've seen, but language and tool complexity is.
→ More replies (0)
1
u/flukus Jan 29 '19
Can anybody explain how modules differ from libraries (static or shared)? They seem to cover the same ground and splitting a project into libraries would solve most of the problems modules seem to.
2
Jan 29 '19
Currently if you want to use a library in your C++ application, you
#include "some_lib.h"which contains declarations of types and functions from the library. This is a bit fragile, preprocessor state in the importer (#defineetc.) can effect the imported library. And preprocessor state of the imported library can affect importing code.As an example, let's say you are developing an application that relies on
windows.h. Amongst other things, this header defines min and max macros that conflict with std::min and std::max. But these macros are#ifdef'd out if you#define NOMINMAXbefore#include <windows.h>.But now headers you include after
windows.hwill also getNOMINMAX. What if one of these headers relies on the min/max macros? Or vice-versa - if your code uses the min/max macros, but you include a libraries header that#define NOMINMAXbefore you includewindows.h, you'll miss out on those macros and your code won't compile.1
u/flukus Jan 29 '19
Isn't this already handled by namespaces (the hard and soft types)?
Does this mean modules will replace headers?
Couldn't you handle this in a build step, running windows.h through the pre-processor?
2
Jan 29 '19
Macros are handled by the preprocessor, before compilation. They can't be namespaced.
This is just my understanding, but I do not think that modules are currently planned to completely replace headers. Rather they are intended to be used across boundaries between different discrete components of an application (between libraries, and between libraries and executables). I do not actually see why you could not use modules to completely replace headers though.
I don't see how. The preprocessor already runs over
windows.hwhen you include it in your source file. The core problem is that due to how they are designed headers can't be isolated from the code that includes them.1
u/flukus Jan 29 '19
The macro might not be able to be namespaced but the value can, if the define is still needed it can be moved to a private header.
I think you just described a header, it's an interface between boundaries. This might be a better solution, but we'll still have 2 ways to do the same thing.
I was talking about your example where the interaction between different headers produced unintended results. In this case you could run the pre-processor with NOMINMAX defined and use this header for compiling your app. I think this should play nice with the linker, but I'd have to check.
1
Jan 29 '19
These aren't insurmountable flaws with headers, but unfortunately they are problems that generally occur outside your own code. Sure it's possible to use headers in a way that's safe and exports no macros, but this is de-facto not the case. Making these into non-issues relies on library authors and system header authors willingness and ability to do so.
1
u/flukus Jan 29 '19
Making these into non-issues relies on library authors and system header authors willingness and ability to do so.
If they don't do so then can you expect them to create modules? And if modules don't replace headers then won't these issues still exist anyway?
6
u/OneWingedShark Jan 28 '19
If you want modules, similar expressive power to C++, and better safety then I'd recommend looking at Ada.
7
Jan 28 '19
People backing the current modules design have not proven that modules work at scale,
I thought that Google had been using modules via clang modules for their multiple million C++ code base since... 2012 ?
6
u/Yehosua Jan 28 '19
I haven't followed the modules proposals very closely, and I'd hoped that the OP would explain this a bit better, because I had the same question as you. However, based on my limited understanding, the sentence after what you quoted does address this:
Existing “modules” deployments do not use the current design, and do not use the automated module scanning that would be required for use with real build systems in the wild.
Google's own modules implementation is part of Clang. Clang's docs have a very "work-in-progress" feel, but if I understand them correctly, it requires that you write module maps yourself, instead of doing the source scanning that the OP is arguing is so bad for performance. (In other words, Clang already does something a bit like number 2 of the OP's "Possible Solutions.")
2
u/mathstuf Jan 28 '19
But are Clang modules compatible with the module proposal in front of the committee? AFAIK, they're not the same thing…
2
Jan 28 '19
Given that the author/maintainer of "Clang modules" co-wrote the latest "C++ modules" proposal, implements it in Clang, and works for Google which has a big code base using "Clang modules". I'd guess they are being paid to work on this by Google to make sure C++ modules work for them.
2
u/lrem Jan 28 '19
And if they work for Google, I guarantee that parallelism has been thought of. Source: I stare at builds spanning thousands C++ files daily at Google. Compilation is blazing fast (but linking hurts).
1
Jan 28 '19
FWIW everybody should take things with a grain of salt, just because "Google" does not imply that things are flawless. People should read the proposals, try the implementations, and form an informed opinion. Still, that takes time that a lot of people might not have, and it is ok to be skeptic about the proposals. As many people have mentioned, with such a large change, and given the amount of C++ users in the world, most of them doing completely different things, it's pretty much impossible that something designed by 50-100 people can be good for all of them. That's just how things are. This does not mean that the people working on this "don't have any idea of what they are doing", only that they can't know everything, but nobody can.
1
u/lrem Jan 28 '19
Yeah, but the author says how this will hurt compile speeds due to breaking parallelism. That's pretty much the one thing that I'm pretty sure the Google engineers involved heavily optimised for.
1
Jan 28 '19
Google uses "Clang Modules" with are different from the "Modules TS". Clang does not fully implement the "Modules TS" so I doubt google has switched to using it.
"Clang Modules" have explicit module map files, which the author mentions as don't have any of the performance problems that the author mentions.
1
u/mathstuf Jan 28 '19
One thing that Google generally (and, IME, pretty consistently) doesn't care about is external dependency consumption. I see this reflected in the "no support" for having modules be an out-of-build artifact…something that'd be really nice for the wider C++ community. Instead, Boost being modularized is damn near useless since if you're not bundling it, you'll "never" have modules for it…unless you write rules into your build system to generate them from whatever MIU might be available from a Boost installation (assuming that is supported across a variety of compilers…). Yay. Just what I want to maintain.
2
u/WalterBright Jan 28 '19
Back in the 1990s, I implemented precompiled headers for Digital Mars C++. I noticed that this created a de-facto module system for C++.
4
Jan 28 '19 edited Jan 28 '19
Clang modules is much clever than PCH, yet it doesn't really perform much better in practice. The main reason is templates.
In C++, there is very little that you can't cache about them easily. E.g. you can't type check them and cache that, you can cache template instantiations but properly implementing invalidation is hard because function overloading, ADL, adding a explicit specialization, etc. can invalidate the template instantiation and users can add these to pretty much any files.
So basically the only thing that you can cache about templates is parsing the files, which is what PCH already does :D
Add to this that pretty much every C++ program uses templates everywhere even if users don't write any manually (
std::vector,<algorithm>,<functional>,<utility>,<variant>,<memory>, ... the standard library exposes a lot of template-based APIs).Imagine you call some std range algorithm from the Ranges TS in some user defined type twice, and in between you define a
beginfunction. In both cases, differentbegins might be picked, so you can't reuse the cached one.
2
u/pezezin Jan 29 '19
How will modules relate to namespaces?
One thing I like about C++ is that you can split a namespace into as many files as you want, to better organize your code. I don't like when a language forces the idea that one module is one file.
1
u/SatansAlpaca Jan 28 '19
With Swift, every file that is included in the build is passed to the compiler, and one file is named the “current file”. The compiler does a first pass on all the files to extract interfaces without parsing definitions and caches the result of that. It then actually compiles the “current file” to an object file. Seems to me that this could be applicable here too.
-3
u/shevy-ruby Jan 28 '19
So not only does the committee worship Cthulhu and add more complexity - they are also incompetent and have no real idea what they are doing anymore.
5
u/ForgedBanana Jan 28 '19
It was always like that. Honestly i think those guys have an irrational hate of the KISS principle.
5
u/Valmar33 Jan 28 '19
I don't understand the downvotes.
Because the increasing unnecessary complexity is one of the reasons why C++ is becoming slowly less popular, and why Rust and Go even became a thing in the first place.
All of C++'s technical debt and backwards compatibility desires really cripple each new major feature added on.
C++ modules... just won't work sanely the way they're currently proposed. They'll just be mostly ignored in favour of headers.
1
u/SatansAlpaca Jan 28 '19
Regardless of how modules work within a single project, IMO, one of the biggest wins will be when it comes to importing external libraries.
-24
Jan 28 '19 edited Jan 28 '19
I like how C++ is slowly becoming Python.
edit: your downvotes cannot stop the transformation.
22
u/kuikuilla Jan 28 '19
By getting a module system?
17
u/ISvengali Jan 28 '19
Next up theyre adding a single global lock that all threads have to take to run, std::thread_safe.
13
u/r0b0t1c1st Jan 28 '19 edited Jan 28 '19
You say,
And implicitly go on to conclude that this prevents them being processed in parallel.
It's not clear to me why the first statement in this line of reasoning has to be true. Can
foo.bmiinstead be a direct product of compilingfoo.hpp? In that case, the comparison is then between:in parallel:
compile
foo.cppwhich includesfoo.hppcompile
bar.cppwhich includesfoo.hppin parallel:
compile
foo.cppwhich includesfoo.hppin sequence:
compile
foo.hpp->foo.bmicompile
bar.cppwhich doesn't seem like a net change in parallelization to me.