Incubating C++ support questions

[Steven Walters]

Hi,

I came across
https://github.com/gradle/gradle/blob/master/design-docs/continuous-delivery-for-c-plus-plus.mda
couple weeks ago and it has piqued my interest.

I've been working on converting some projects to use gradle at work as
we're primarily java though given that we tend to deal with some
applications that only have C/C++ APIs.
Honestly, converting these projects have become a bit "clunky" with needing
to utilize something such as GNU make for the C++ portions.

Overall, the series of stories is fairly well thought out, but I did have
some questions.
(This did come out to be rather long, so maybe tackle this in 'chunks'?)


1) Regarding the 'Build a static library binary' story, there is a point
listed
"Add the '-fPIC' flag when compiling to ensure that the static library can
be included in a shared library"
That I have some concerns with making this a forced option.
There are typically performance implications caused by PIC-supported native
code that projects have some tendency to avoid when only building static
libraries/executables.

I would argue that this should not be a forced option, but instead be more
dynamic in being enabled only if shared code is configured to also be built
and then have static-only builds add the option/flag by the user if desired.

The natural exclusion to this is Windows, where all code has to be PE,
which makes the -fPIC option cause an ignore warning in MinGW, due to it
always having to make PE-compatible code.


2) Regarding the 'Build a native component using multiple tool chains' and
'Build a native component for multiple architectures' stories,
Is there any plan to switch to the Visual Studio discovery mechanism to be
more dynamic than expecting the cl, lib and link executables to already be
on the Path (meaning that the environment was already configured)?

we often find ourselves needing to build a library using multiple versions
of Visual studio, such as 8 (2005), 9 (2008), and 10 (2010)... soon this
will likely include 11 (2012).
There are various reasons for this, primarily stemming from the the
application running our libraries not allowing a mixture of MSVC runtimes
or STL incompatibilities between different versions of MSVC (this is
similar in nature to libstd++ major version differences)

I would ignore the existence of MSVS versions 7.0 (2002) and 7.1 (2003) due
to their age, somewhat differing folder structure from the later versions,
and lack of x86_64 support.


3) Again regarding 'Build a native component using multiple tool chains',
I am currently wondering if Intel's compiler would warrant official
recognition, or be left to the user to configure the binary names and flags
themselves...
There are some differences between MSVS and Intel's Compiler for Windows
that could be benefit from this, such as Intel's compiler having -QMM/-QMT
options that work similar to gcc/g++'s -MM/-MT flags, though I've found
this to be fairly slow in comparison to MinGW.
It will also be important when it comes to the story where MSVS
projects/solutions would be created.
I'm not aware of much difference on Linux-based or Mac OSX versions of
Intel's compiler outside of the toolchain binary names that would require
official recognition.


4) pkg-config as a 'repository'?
I did not see mention of pkg-config in the series of stories.
I'm not sure if it was meant to be implied within the publishing or
dependency stories, but given how it works pretty close to a repository
(though a local one) and fairly prevalent among the unix-based systems, it
would be beneficial to see it be supported within gradle.


5) Profile optimizations?
Visual Studio and GCC both have concepts of optimizing binaries through
executable profiling.
This makes building a good bit more complicated as you technically have to
build the executable with profiling enabled, execute it some <X> number of
times, and then rebuild it again with the profiling data.
Naturally, this can't work for cross-compilation scenarios, but it may be
worth to have as an additional story at the end.


I also have interest in contributing to the completion of the support for
C++ and associated languages, though I'm not sure how much 'ramp up time'
there would be due to my current unfamiliarity with gradle's overall
codebase.

Thank you for spending the time to read this rather long email, and I look
forward to any feedback.
--
Steven Walters

[Adam Murdoch]

Hi,
I came across https://github.com/gradle/gradle/blob/master/design-docs/continuous-delivery-for-c-plus-plus.md a couple weeks ago and it has piqued my interest.
I've been working on converting some projects to use gradle at work as we're primarily java though given that we tend to deal with some applications that only have C/C++ APIs.
Honestly, converting these projects have become a bit "clunky" with needing to utilize something such as GNU make for the C++ portions.
Overall, the series of stories is fairly well thought out, but I did have some questions.
(This did come out to be rather long, so maybe tackle this in 'chunks'?)
1) Regarding the 'Build a static library binary' story, there is a point listed
"Add the '-fPIC' flag when compiling to ensure that the static library can be included in a shared library"
That I have some concerns with making this a forced option.
There are typically performance implications caused by PIC-supported native code that projects have some tendency to avoid when only building static libraries/executables.
I would argue that this should not be a forced option, but instead be more dynamic in being enabled only if shared code is configured to also be built and then have static-only builds add the option/flag by the user if desired.

I think we'll end up not using -fPIC by default. Instead, we'll solve this by combining a few things:

1. Remember whether a given static library binary was built as relocatable code or not. This will travel with the binary as part of its meta-data.
2. Have dependency resolution understand that at link time, only a relocatable binary can be linked into a relocatable binary.
3. Make it easy to produce either or both relocatable and non-relocatable static library variants, for those toolchains and platforms where it makes sense.

The natural exclusion to this is Windows, where all code has to be PE, which makes the -fPIC option cause an ignore warning in MinGW, due to it always having to make PE-compatible code.

This would be baked into #1 and #3, above.

2) Regarding the 'Build a native component using multiple tool chains' and 'Build a native component for multiple architectures' stories,
Is there any plan to switch to the Visual Studio discovery mechanism to be more dynamic than expecting the cl, lib and link executables to already be on the Path (meaning that the environment was already configured)?

Absolutely. There will probably be a few parts to this:

1. Change the discovery mechanism to look in the default visual studio install locations. We currently do this in our integration test fixtures, so we'd move this code into the plugin.
2. Add some way to specify where to look for various versions of a tool chain.
3. Handle multiple versions of the same toolchain. At the moment, there's just 'visual c++' or 'gcc' but no way to specify which versions are required, or to build using multiple different versions.
4. For the visual c++ tool chains, automatically set up all the various environment variables that need to be set when invoking the command-line tools.

we often find ourselves needing to build a library using multiple versions of Visual studio, such as 8 (2005), 9 (2008), and 10 (2010)... soon this will likely include 11 (2012).
There are various reasons for this, primarily stemming from the the application running our libraries not allowing a mixture of MSVC runtimes or STL incompatibilities between different versions of MSVC (this is similar in nature to libstd++ major version differences)
I would ignore the existence of MSVS versions 7.0 (2002) and 7.1 (2003) due to their age, somewhat differing folder structure from the later versions, and lack of x86_64 support.
3) Again regarding 'Build a native component using multiple tool chains',
I am currently wondering if Intel's compiler would warrant official recognition, or be left to the user to configure the binary names and flags themselves...
There are some differences between MSVS and Intel's Compiler for Windows that could be benefit from this, such as Intel's compiler having -QMM/-QMT options that work similar to gcc/g++'s -MM/-MT flags, though I've found this to be fairly slow in comparison to MinGW.
It will also be important when it comes to the story where MSVS projects/solutions would be created.
I'm not aware of much difference on Linux-based or Mac OSX versions of Intel's compiler outside of the toolchain binary names that would require official recognition.

We do want to make the toolchain APIs pretty extensible, so that it is relatively easy (or at least possible) to add custom implementations. It should be possible to implement a plugin that can add support for additional compilers (the GCC and Visual C++ integrations are implemented as plugins already, they just happen to be automatically applied for you).

We'll probably let the community drive exactly which compilers and versions that we officially support, so that if someone is willing to implement and help support the Intel compiler integration, we'll be happy to include it.

Same with support for other platforms, too. We'll need some help from the community with integrating with and supporting specific operating systems and architectures.

4) pkg-config as a 'repository'?
I did not see mention of pkg-config in the series of stories.
I'm not sure if it was meant to be implied within the publishing or dependency stories, but given how it works pretty close to a repository (though a local one) and fairly prevalent among the unix-based systems, it would be beneficial to see it be supported within gradle.

Absolutely. This is a good idea. I've added a brief mention of this to the spec.

The aim is to produce native libraries and applications packaged in a number of different formats, including things like a simple zip or tar, an OS X bundle, an installer, an RPM or DEB or pkg-config package, a NuGet bundle, as an 'installation' (eg installed into /usr/local/lib and /user/local/bin), or as a source distribution. Or ...

Beyond building these packagings, each type of packaging should be publishable to and resolvable from various types of repositories. Some repositories will accept only a certain type of packaging and others will be more general purpose and accept any type of packaging.

5) Profile optimizations?
Visual Studio and GCC both have concepts of optimizing binaries through executable profiling.
This makes building a good bit more complicated as you technically have to build the executable with profiling enabled, execute it some <X> number of times, and then rebuild it again with the profiling data.
Naturally, this can't work for cross-compilation scenarios, but it may be worth to have as an additional story at the end.

Excellent idea.

I also have interest in contributing to the completion of the support for C++ and associated languages, though I'm not sure how much 'ramp up time' there would be due to my current unfamiliarity with gradle's overall codebase.

You'd be most welcome to contribute. Is there something specific you were interested in?

We're currently deeply reworking the C++ support, to implement a bunch of the stories in the spec. The Gradle 1.7 release will contain the first round of this work, with more to follow in the next few releases. So, the internals (and externals) will be changing quite a bit, which might make it a bit tricker to get started.

If you let us know what you're interested in working on, we can help out by fleshing out the spec with some implementation hints and so on.

Thank you for spending the time to read this rather long email, and I look forward to any feedback.

Thanks for your excellent feedback. Much appreciated.


--
Adam Murdoch
Gradle Co-founder
http://www.gradle.org
VP of Engineering, Gradleware Inc. - Gradle Training, Support, Consulting
http://www.gradleware.com

Join us at the Gradle Summit 2013, June 13th and 14th in Santa Clara, CA: http://www.gradlesummit.com

[Russel Winder]

On Mon, 2013-06-10 at 10:38 +1000, Adam Murdoch wrote:
[
]

Post by Adam Murdoch
We'll probably let the community drive exactly which compilers and
versions that we officially support, so that if someone is willing to
implement and help support the Intel compiler integration, we'll be
happy to include it.
Same with support for other platforms, too. We'll need some help from
the community with integrating with and supporting specific operating
systems and architectures.

I think the obvious set to have as a target is (cl, gcc, clang, intel)
on (Windows, OS X, Linux, FreeBSD). Although not all may be created by
Gradleware folk, the "holes" awaiting community input should be clear.

SCons and Waf are the competition to keep a view on.

[Adam Murdoch]

Post by Russel Winder
[…]

Post by Adam Murdoch
We'll probably let the community drive exactly which compilers and
versions that we officially support, so that if someone is willing to
implement and help support the Intel compiler integration, we'll be
happy to include it.
Same with support for other platforms, too. We'll need some help from
the community with integrating with and supporting specific operating
systems and architectures.

I think the obvious set to have as a target is (cl, gcc, clang, intel)
on (Windows, OS X, Linux, FreeBSD).

This is a pretty good list. Windows, OS X and Linux are officially supported (exact versions TBD). The plugins work on FreeBSD and Solaris as well, and probably on other platforms too, but we don't have CI coverage for these, so they're not on the official list yet.

Compiler-wise, cl and gcc are officially supported (again, exact versions TBD). I also have some clang integration ready to merge, I just need to find time to add some CI coverage for it. There's no support for the intel compiler at the moment.

Post by Russel Winder
Although not all may be created by
Gradleware folk, the "holes" awaiting community input should be clear.

I think that's true. We're certainly hoping that the community can help out with some of the missing integrations.


--
Adam Murdoch
Gradle Co-founder
http://www.gradle.org
VP of Engineering, Gradleware Inc. - Gradle Training, Support, Consulting
http://www.gradleware.com

Join us at the Gradle Summit 2013, June 13th and 14th in Santa Clara, CA: http://www.gradlesummit.com

[Russel Winder]

On Tue, 2013-06-11 at 15:54 +1000, Adam Murdoch wrote:
[
]

Post by Adam Murdoch
I think that's true. We're certainly hoping that the community can help out with some of the missing integrations.

[
]

Perhaps an element of helping things along rather than awaiting
volunteers might be in order. I know some folks at Intel associated with
the periphery of the Intel C and C++ compilers, perhaps they might be
convinced of the marketing benefit to ensuring that the Intel compiler
is supported in Gradle? Unless Gradleware folk are already talking with
Intel of course.

[Steven Walters]

Post by Steven Walters
2) Regarding the 'Build a native component using multiple tool chains' and
'Build a native component for multiple architectures' stories,
Is there any plan to switch to the Visual Studio discovery mechanism to be
more dynamic than expecting the cl, lib and link executables to already be
on the Path (meaning that the environment was already configured)?
1. Change the discovery mechanism to look in the default visual studio
install locations. We currently do this in our integration test fixtures,
so we'd move this code into the plugin.
2. Add some way to specify where to look for various versions of a tool chain.
3. Handle multiple versions of the same toolchain. At the moment, there's
just 'visual c++' or 'gcc' but no way to specify which versions are
required, or to build using multiple different versions.
4. For the visual c++ tool chains, automatically set up all the various
environment variables that need to be set when invoking the command-line
tools.

Given that most of the installation information of Visual Studio is in the
registry (naturally), outside of looking at the default locations, the
environment variables VS<#>COMNTOOLS can be used as a basis for locating a
given version of an installation.
typically going up 2 parent directories of this environment variable leads
to the root install directory, from there you can then get to the
VC\vcvarsall.bat and use a 'x86' or 'x86_amd64' argument string to
initialize the environment.
Though some care is likely to be needed here as the x86_64 compiler has a
tendency to be an optional installation piece, so it might not always be
there.
This is a practice that we've been using to support building in multiple
versions.

Post by Steven Walters
I also have interest in contributing to the completion of the support for
C++ and associated languages, though I'm not sure how much 'ramp up time'
there would be due to my current unfamiliarity with gradle's overall
codebase.
You'd be most welcome to contribute. Is there something specific you were interested in?
We're currently deeply reworking the C++ support, to implement a bunch of
the stories in the spec. The Gradle 1.7 release will contain the first
round of this work, with more to follow in the next few releases. So, the
internals (and externals) will be changing quite a bit, which might make it
a bit tricker to get started.
If you let us know what you're interested in working on, we can help out
by fleshing out the spec with some implementation hints and so on.

That's a bit of a tough question, given that I'm not aware of how
aggressive of a timeline in that all the milestones/stories are trying to
get hit.

Between working on/with open source projects in such as x264 and ffmpeg
primarily on a windows environment with MinGW and just about every version
of MSVS at my work,
I'd probably say the above is where my knowledge is primarily located.

Outside of the visual studio project/solution integration stories, not
currently sure where else I would be useful,
though depending on what kind of timeline is being targeted for the
completion of all this work I could potentially grab some other bits, time
allowing.

Thanks,
---
Steven Walters

Post by Steven Walters
--
Adam Murdoch
Gradle Co-founder
http://www.gradle.org
VP of Engineering, Gradleware Inc. - Gradle Training, Support, Consulting
http://www.gradleware.com
http://www.gradlesummit.com